Disputes and Appeals

Programmes treat disputes as friction to be minimised. A functioning dispute process is the main source of evidence about whether the scoring works.

Why it matters more than it appears

Disputes locate broken items. An item disputed frequently and upheld frequently is badly written.

They locate inconsistent evaluators. A pattern by evaluator is visible in the dispute log and nowhere else.

They locate transcription failures, which is critical where automated scoring is used and where the accuracy gap by accent is a live concern.

They are the safety valve. A programme with no dispute route produces resentment that surfaces as disengagement rather than as fixable feedback.

A dispute rate of zero is a warning, not a success. It means the route does not work or agents have concluded it is pointless.

Designing the process

The agent can see the evidence. The call, the transcript, the item, the reasoning. A dispute against a score they cannot examine is not a dispute.

A defined route and a defined timeframe.

Reviewed by someone other than the original evaluator. Obvious, and frequently not the case.

An outcome recorded with a reason, on both upheld and rejected disputes.

No consequence for disputing. If raising one is a career risk, the mechanism is decorative.

What to do with the log

This is the part almost nobody does.

Dispute rate by item. High rates indicate a definition problem.

Upheld rate by item. High rates indicate the item is wrong.

Dispute and upheld rate by evaluator. A pattern is a calibration matter.

Upheld rate by agent group, including by accent or language where lawful and relevant. A systematically higher upheld rate for a group is evidence of unequal accuracy and it is exactly the finding you need.

Review quarterly. Every upheld dispute is a defect report on the scorecard, and treating them as individual cases rather than as data wastes the signal.

Automated scoring specifically

Where a system scored, the dispute process carries more weight.

The agent must be able to hear the audio and read the transcript together. Most disputes about automated compliance findings resolve immediately when the two are compared, and the resolution is usually a recognition error.

Track the proportion of automated findings overturned. Above a threshold, the item should not be scored automatically.

Route recognition errors back to the vocabulary configuration. A term consistently misheard should be added, and the dispute log is where you find out which.

The escalation route

Beyond the immediate review, a route to someone independent of the operational line.

Necessary where scores feed pay or performance management, and increasingly expected under employment practice.

Rarely used and important that it exists.

The cultural signal

How disputes are handled tells agents what the programme is for.

A process that upholds disputes when they are right demonstrates the programme wants to be accurate.

A process that never upholds anything demonstrates it wants to be unchallenged, and everyone draws the obvious conclusion.

Publishing the upheld rate internally is uncomfortable and it is the clearest possible statement that the programme is measuring itself as well as the agents.

The quarterly dispute review

Thirty minutes with the dispute log, producing changes rather than case outcomes.

Dispute rate by item. The top three are candidates for rewriting.

Upheld rate by item. Anything above a threshold means the item is wrong, not that the agents are lucky.

Dispute and upheld rate by evaluator. A pattern is a calibration conversation.

Upheld rate by agent group, where lawful and meaningful. A disparity here is the accuracy gap made visible.

Time to resolution, because a dispute route that takes six weeks is not used.

Disputes raised as a proportion of evaluations, trended. Falling toward zero means agents have given up.

Produce two or three specific changes from each review — a rewritten item, a recalibration, a vocabulary addition — and check them at the next one.