There is no single accuracy number that tells you whether your casino support is good. Published figures exist, but most measure product questions in other industries, not responsible gambling, withdrawals or VIP handling. A useful benchmark set for a licensed operator combines factual accuracy, escalation quality, compliance and human review metrics, each defined against your own policies.

This guide sets out what to measure, which public figures are worth knowing (and how far they travel), and how to build an internal benchmark you can defend to a regulator.

TL;DR

Public accuracy figures mostly come from general customer service. Treat them as context, not as targets for a casino.
Measure several things at once: factual accuracy, correct escalation, policy compliance, reviewer override rate and response time.
Under UK rules, automated detection of harm must be backed by manual review, so human oversight metrics are part of accuracy, not an add-on.
Build your own test set from real tickets, re-run it whenever a prompt, model or knowledge base changes, and log every decision.
Review 100% of AI drafts before they go out in high-risk tiers, and track what reviewers change.

Why a single accuracy score misleads
What the public numbers say, and where they stop
The metrics that make up a casino support scorecard
What UK rules mean for your targets
Building your own benchmark in five steps
Measuring the humans too
Our view: accuracy is a process, not a score
Where Pitbot fits
A worked example of a benchmark review
Common mistakes
FAQ
Sources

Why a single accuracy score misleads

Accuracy sounds simple: the share of answers that were right. In practice it hides most of what matters in a casino inbox.

A reply can be factually correct and still wrong. It might quote a bonus term accurately to a player who has just shown signs of harm. It might answer a withdrawal question correctly but miss that the player is asking for the third time and is angry. Neither failure shows up in a headline percentage.

Researchers writing on AI evaluation make a related point: a reported benchmark score is often an inference about capability rather than a direct measure of it, so evaluation should be grounded in a specific idea of what the system is meant to do in your setting. For a casino operator, that idea is not “answer questions”. It is “answer questions correctly, in line with licence conditions, and hand over to a person when the situation calls for it”.

What the public numbers say, and where they stop

One 2026 comparison reports that purpose-built customer service AI tools reach roughly 95.1% to 97.8% accuracy on product queries, against 71.8% to 72.3% for general-purpose models, which also showed higher hallucination rates of 5.9% to 8.4%. The gap is real and the direction is useful: a system grounded in your own knowledge base beats a general chatbot guessing from training data.

Three cautions before you quote those numbers in a board paper:

They measure product queries, usually in software or retail. Casino support adds bonus terms, wagering requirements, KYC, self-exclusion and complaints.
They test a fixed set of questions. Your players write differently, in several languages, often upset and often mid-session.
They say nothing about what happens to the remaining 2% to 5% of answers. In a regulated market, the handling of those is the point.

Contact centre standards take the same line. ISO 18295 is generally described as a framework in which the organisation defines its own indicators rather than one that imposes fixed numerical targets. That is a sensible reading for casino support too: the benchmark is the one you set, document and meet.

The metrics that make up a casino support scorecard

A balanced scorecard has four layers. Each answers a different question.

Layer	Metric	What it tells you
Content	Factual accuracy against your knowledge base	Is the answer right?
Content	Hallucination rate (claims not in any source)	Is the system inventing terms or policies?
Judgement	Correct escalation rate	Did sensitive cases reach a person?
Judgement	Missed-escalation rate	Which cases should have been escalated and were not?
Compliance	Policy adherence (tone, disclosures, marketing suppression)	Would this reply survive an audit?
Oversight	Reviewer override rate	How often do staff change or reject drafts?
Speed	First response and time to resolution	Is the player waiting?

Factual accuracy

Score replies against a written source of truth: your terms and conditions, payment policy and help centre. Mark each reply correct, partly correct or wrong, and record why. “Partly correct” matters: an answer that omits the wagering condition is not a pass.

Escalation accuracy

This is the metric most generic benchmarks leave out, and the most important one for a casino. Build a labelled set of conversations that should escalate: signs of problem gambling, complaints about fairness, threats of chargebacks, bereavement or distress, VIP requests with financial weight. Then measure two things: how many the system flagged, and how many it missed. The missed rate is the one to drive towards zero, even at the cost of more false alarms.

Compliance adherence

Check replies against a checklist your compliance officer owns. For example: no promotional language to a player flagged for harm, correct jurisdiction wording, no promises about outcomes. Our guide to human-reviewed AI for regulated markets sets out the oversight metrics we think matter, and recommends tracking them on a fixed cadence, weekly at minimum for high-risk tiers.

Override rate

If reviewers change 40% of drafts, accuracy is poor. If they change 0%, you should ask whether they are reading at all. Regulators and auditors look for patterns of automation bias and human disengagement, and those patterns show up when you aggregate audit logs, so the log should be queryable by override rate and not only case by case.

What UK rules mean for your targets

For UKGC licensees, accuracy cannot be separated from oversight. The Gambling Commission’s guidance on automated processes for strong indicators of harm says licensees must act on those indicators in a timely manner by using automated processes, and where such processes are applied, they must also manually review them. The guidance was published in 2022, so check the current version before you rely on it. The licence condition itself, LCCP 3.4.3 on remote customer interaction, also requires licensees to prevent marketing and new bonus offers where strong indicators of harm have been identified.

Translated into benchmarks:

Time to action: measure the gap between a harm indicator appearing and a documented action.
Marketing suppression accuracy: measure how many flagged players received any promotional message. The target is none.
Manual review coverage: record what share of automated harm decisions a person reviewed, and how quickly.

The Dutch regulator has taken a similar line on chatbots and AI. Coverage of its letter points to transparency, meaning the operator must be able to understand and explain why a system acts as it does. If you also hold licences elsewhere, an explainable decision log is a benchmark in itself.

Building your own benchmark in five steps

Five steps for building a support benchmark

1. Collect a representative sample

Pull a few hundred real tickets across channels (email, chat, SMS, Telegram) and languages, anonymised. Include the awkward ones: angry, ambiguous, multi-issue. A set of only easy FAQs will flatter any system.

2. Write the answer key

For each ticket, have an experienced agent and a compliance reviewer write what a correct reply contains and whether it should escalate. Where they disagree, that is a policy gap worth fixing before you measure anything.

3. Diagnose failures by layer

Do not reach for a bigger model by reflex. Practitioners argue for a diagnosis-first approach: work out whether failures originate in retrieval, reranking or the final decision, because fixing the evidence layer often gains more than swapping the model. In support terms: was the right policy page found, was it ranked first, and did the reply use it correctly?

4. Set confidence thresholds, then keep recalibrating

If you filter drafts by model confidence, remember that confidence distributions are sensitive to prompt changes. One published support case study on token log-probabilities as a hallucination filter shows the idea; the practical lesson is to recalibrate thresholds whenever the system prompt changes.

5. Re-run on every change

Treat the test set like a regression suite. Re-run it when you change the model, the prompt, the knowledge base or a bonus policy. Keep dated results so you can show an auditor the trend.

Measuring the humans too

A human-in-the-loop model is only as strong as the reviewers. Two measures are worth adding to the scorecard.

Reviewer competency. Document it through periodic re-testing, not a one-off induction. Auditors want proof that reviewers understand the specific AI system they supervise.

Edit categories. When a reviewer changes a draft, tag why: factual error, tone, compliance, missing escalation. Over a month, those tags tell you where the AI is weak and where your knowledge base is thin. Our piece on human-in-the-loop AI for iGaming player support covers how this review step fits into a support workflow.

Our view: accuracy is a process, not a score

We build Pitbot for online casino operators, and we start from a position that may sound conservative: keep a person in the loop where it matters. With auto send off, AI-drafted replies on email, live chat, SMS and Telegram wait for staff approval before they reach a player, and responsible gambling and regulator flags on those channels are a hard stop: the thread is escalated to a person instead of answered, and a sensitive case opens for someone to pick up. Speed and compliance pull in different directions, and a person in the loop is how you get both.

That shapes how we think about benchmarks. A model that is right 97% of the time still sends a wrong answer to someone, and in casino support the wrong answer may go to a vulnerable player. So we look at what happens around the model: whether sensitive cases are escalated, whether marketing is suppressed for flagged players, whether the approver of each signed-off reply is recorded.

Our view: accuracy is a process, not a score — overview diagram

We also think operators should be wary of any vendor who offers one headline accuracy figure. Ask what was tested, in which languages, on whose tickets, and what the failure handling looked like. Ask for the audit trail behind the number.

The benchmark we would put first is the missed-escalation rate, followed by override rate. Both measure judgement, and both are things you can only see if every decision is logged.

— pitbot

Where Pitbot fits

If you want these measures without building the tooling yourself, Pitbot combines multi-provider AI drafting with the option to hold replies for staff approval, across email, live chat, SMS and Telegram. It includes sentiment tracking and responsible gambling escalation, thread assignment to a named owner with handover history, responsible gambling escalation with marketing blocked while a sensitive case is open, append-only audit trails and operational analytics in one dashboard. See the Pitbot features for detail, or the page on AI customer support for online casino operators if you are mapping it to your support team.

A worked example of a benchmark review

Take a hypothetical operator reviewing a month of 500 sampled conversations. The numbers below are illustrative only, to show how to read a scorecard, not results from any client.

470 replies are factually correct. That looks like 94%, which is respectable.
Among the 30 wrong replies, 22 were caught by reviewers before sending. Eight reached a player, so the delivered error rate is 1.6%, and the review step is what kept it there.
Of 20 conversations that should have escalated, 18 did. Two were missed, a 10% missed-escalation rate. This is the figure that needs a root-cause review, whatever the headline accuracy says.
Reviewers edited 120 drafts, a 24% override rate. Tagging shows most edits were tone, not facts, which points to persona and prompt tuning rather than the knowledge base.

Notice how the story changes at each layer. The headline number says “fine”; the escalation number says “investigate”.

Common mistakes

Benchmarking on FAQs only. Easy tickets inflate scores.
Averaging across languages. A strong English score can hide weak Turkish or German replies. Report per language.
Ignoring the tail. One missed self-exclusion request outweighs a hundred correct bonus answers.
Comparing with other industries’ CSAT. Satisfaction scores from general support do not reflect casino contexts where a correct refusal may annoy a player.
Treating the benchmark as finished. Policies, bonuses and models change, so the test set must too.

FAQ

What is a good accuracy rate for AI casino support?

There is no regulator-set figure. Published comparisons show purpose-built support AI reaching the mid-to-high 90s on product questions, but those are not casino-specific. Set a target on your own test set, and put more weight on missed escalations and compliance failures than on the headline rate.

How often should we re-test?

Re-run your test set whenever the model, prompt, knowledge base or key policy changes. For oversight metrics such as override rate, review weekly at minimum for high-risk tiers.

Can AI handle responsible gambling conversations alone?

Not under UK rules as we read them. The Gambling Commission’s guidance requires manual review where automated processes act on strong indicators of harm. AI can help detect and draft, but a person should review and stay available to intervene.

Is staff approval of AI replies too slow?

It adds a step, but drafts arrive ready for approval, so staff are editing rather than writing from scratch. Track time to first response alongside override rate to see whether the balance is right for your volumes.

Support Accuracy Benchmarks for AI in Online Casinos