Quality and team

Running QA Calibration Sessions That Keep Scoring Fair

How to run calibration sessions so reviewers score calls the same way: choose calls, score blind, discuss gaps, refine the scorecard and track agreement.

Call editorial team · Published · 6 min read

If two reviewers listen to the same call and one scores it very good while the other fails it, agents quickly stop trusting the scores. Calibration is the habit that stops this. Reviewers score the same calls separately, compare notes and settle the differences, so that a score means the same thing whoever gave it.

A small group of quality reviewers discussing a printed scorecard around a meeting table
Illustrative image

This guide shows how to run calibration sessions in a small team. It builds on Quality Management for Calls and supports the feedback loop in A Simple Framework for Coaching Agents on Calls.

Why calibration matters

Scorecards are written in words, and words get read differently. "Showed empathy" or "resolved the issue" mean different things to different reviewers, or to the same reviewer on different days. Without calibration, you get three problems: unfair scores between agents, coaching based on opinion, and reports that cannot be compared across time or teams.

Signs that you need calibration

  • Agents dispute scoresThey say a different reviewer would have passed the same call.
  • Reviewer averages differOne reviewer is consistently harsher or kinder than another.
  • New reviewers driftScores shift when someone joins or covers.
  • Scorecard items are vagueDifferent people explain them differently.

Preparing a session

A good session needs a little preparation. The facilitator should be someone who knows the scorecard well and can stay neutral, often a QA lead or senior team lead.

Before the session

  1. Choose three or four callsInclude a clear pass, a clear fail and at least one borderline call.
  2. Remove identifying detailsHide agent names, so the discussion stays about the call.
  3. Share the scorecardEveryone uses the current version, with definitions.
  4. Ask for independent scoringEach reviewer scores before the meeting, and does not talk to the others about it.

Real calls give the best material. Pull borderline calls from your normal review, because they are the ones where disagreement hides. Check that the recordings can be shared with the reviewers according to your internal rules, and handle any personal data carefully; see Call Recording Consent for the recording side of this.

Running the session

Start with the scores, not opinions. Put everyone's item-by-item results on a shared screen and look for the items with the widest gaps. Work through those first.

  1. State the spread. For each call, show the total score of every reviewer and the item-level differences.
  2. Ask each reviewer to explain a disputed item, quoting the moment in the call that drove their decision.
  3. Return to the scorecard wording. Does it say clearly what counts as a pass?
  4. Agree the answer for this call, and write down the reasoning.
  5. If the wording is the problem, note the change to make; do not fix it live in a way that confuses people.

Ground rules

A hypothetical session in practice

Imagine a six-person team with three reviewers. They pick four calls and score them alone. On the third call, one reviewer gives a pass on the item about confirming the customer's details and another gives a fail. The facilitator asks each to play the moment they relied on. It turns out the agent confirmed the postcode but not the name, and the scorecard simply says confirm details. The team agrees the item should list exactly which two details count, rewrites it and records the decision.

That small change removes a source of unfair scores across every future review. This is the pattern to look for: a disagreement about a call is usually a sign of a gap in the scorecard.

Remember to close the loop with agents. Tell them what changed in the wording and why, and apply the change from a stated date, rather than rescoring old calls. If a past score was clearly unfair because of the old wording, be open about correcting it.

Turning disagreement into a better scorecard

The most valuable result of a session is usually a clearer scorecard. If reviewers disagree on an item, the item is probably vague. Make it concrete.

Vague itemClearer version
Showed empathyAcknowledged the customer's problem in their own words before offering a solution.
Resolved the issueGave a complete answer or set a clear next step with a time.
Professional toneNo interruptions, no sarcasm, and a polite opening and closing.
Followed processVerified the customer using the steps in the current procedure.

Add one or two example phrases of a pass and a fail to each item. Use "N/A" deliberately for items that do not apply to a call type, rather than guessing. Keep critical items, those that can fail a whole call, to a very small list, and make them unambiguous. A reasonable pass mark is a decision for your team; avoid changing it without telling agents.

Measuring agreement over time

You do not need complex statistics. A simple measure is the spread between the highest and lowest total score for each calibrated call. Track the average spread by month, and look at the items that cause the most disagreement.

Reading the spread

Small spreadLarge spread
What it suggestsReviewers read the scorecard similarlyReviewers read it differently
What to doKeep the routine and rotate callsRewrite items, add examples, run another session soon
RiskComplacency, same blind spotsUnfair scores and mistrust

As a rule of thumb, a spread above about twenty points on a hundred-point scale is a sign that reviewers are using the scorecard differently, and that it is worth agreeing on the wording together. Treat any such number as a guide for your own team, not a standard.

Keep a decision log. A simple shared document with the date, the call, the disputed item and the agreed reading is enough. New reviewers can read it to learn how the team applies the scorecard, and it prevents the same argument from returning every month. Review the log when you revise the scorecard, and retire entries that the new wording makes unnecessary.

Frequency and who takes part

Pair calibration with your sampling plan so reviewers see a mix of agents and call types; see How Many Calls Should You Review?. If an AI tool pre-scores calls, include a few of its results in calibration too, so people learn when to trust it and when to override it.

Calibration in Call

Call's Review & coaching area includes a "Calibration" section that lists calls scored by more than one reviewer, together with the spread between their scores. Admins and managers score calls from the review queue using Pass, Fail or N/A for each scorecard item, and critical items zero the score if they fail. Call's guidance is that a spread above 20 points means reviewers read the scorecard differently, so they should agree on it together.

Call can also score some items on its own where you describe them as words said or not said, using the "Automatic scoring" rules. Keep these rules narrow and true; a blank rule is better than a guessed one. The review queue and scorecard require an active scorecard in your workspace, so set that up before planning calibration around it.

Frequently asked questions

How is calibration different from normal quality reviews?

A normal review has one reviewer score a call. In calibration, several reviewers score the same call independently, then compare. Its purpose is to align the reviewers, not to judge the agent.

Do reviewers have to get identical scores?

No. Small differences are normal. The goal is to agree on what counts as a pass for each item and to keep the overall spread small.

How many calls should we use in each session?

Three or four is usually enough. More calls make the meeting long and shallow. Choose calls that expose disagreement.

Should agents see calibration results?

Agents should see the scorecard definitions and any changes that result from calibration. Whether you share the call-by-call comparisons is a team decision, but openness builds trust.

What if one reviewer is always harsher?

Look at which items drive the gap and discuss them in the session. Often the scorecard wording is unclear. If the gap persists, give that reviewer extra guidance and examples.

How often should the scorecard be updated?

Update it when calibration shows recurring confusion, when your process or product changes, or when customers' needs change. Announce changes to agents.

Compare reviewer scores on the same call in Call

In the demo console you can try incoming calls in a simulation.

Related guides