What should happen when an AI tool fails or is used without authorisation?

When an AI tool fails, the response should protect the client file first

Professional services firms should not treat an AI failure as a minor technology glitch. If a tool gives an unsafe answer, invents detail, loses context, uses the wrong data or appears in a workflow without approval, the firm needs a short, disciplined incident path. The point is to protect the client matter, preserve evidence and show that a qualified person, not the tool, made the final judgement.

The practical test is simple: could the firm explain what happened to a client, regulator, insurer or internal file reviewer without relying on guesswork? If not, the response needs to slow the work down until the facts are clear.

The safest answer is containment, review and recorded remediation

When an AI tool fails or is used without authorisation, pause the affected output, identify the work and data involved, assign a professional owner, and keep an audit trail of the review. Do not send, file, bill, rely on or reuse the output until a competent reviewer has checked the facts, risks and client impact.

For low-risk internal work, remediation may be as simple as correcting the output, logging the event and updating guidance. For client advice, regulated submissions, financial analysis, evidence summaries or confidential documents, the firm should run a fuller review: what data was exposed, what decision might be affected, who approved the correction, and whether the client, insurer or regulator needs to be told.

Review your AI risk controls before incidents repeat

Separate tool failure from unauthorised use

These two problems overlap, but they need different controls. A tool failure means an approved or tolerated system produced something unsafe, misleading or unreliable. Unauthorised use means someone used a tool, account, browser extension, embedded feature or public AI service outside the firm's policy.

For tool failure, the firm should examine quality assurance, prompts, data sources, model settings, retrieval logic, human review and escalation thresholds. For unauthorised use, the firm should examine access control, training, procurement, approved-tool lists, browser controls, supervision and incentives. Staff often bypass policy because the approved route is unclear, too slow or does not fit the actual task.

Both routes should end in a written record. That record does not need to be long, but it should say what happened, which client or internal work was affected, what data was involved, who reviewed the output, what changed, and whether the incident revealed a wider control gap.

Use a practical incident triage checklist

A proportionate AI incident review can start with six questions:

• Scope: Which client, matter, audit, valuation, advice note, report or internal workflow was affected?
• Data: Was confidential, personal, privileged, commercially sensitive or regulated data entered into the tool?
• Reliance: Did anyone rely on the output for advice, analysis, client communication, pricing, evidence, filing or decision support?
• Review: Who is the named professional owner, and what independent check has been completed?
• Disclosure: Does the client, insurer, regulator or internal risk committee need to be informed?
• Prevention: What control changed so the same failure is less likely next time?

This checklist keeps the response anchored in professional duty rather than tool preference. It also helps partners distinguish a harmless drafting issue from a data protection, confidentiality, professional indemnity or client-outcome risk.

Build the governance route before the first serious incident

The best time to design this path is before the first urgent failure. Firms should define approved AI uses, prohibited uses, review standards, disclosure triggers and escalation routes in plain language. The policy should explain what staff should do when they are unsure, because uncertainty is where shadow AI often appears.

Good governance also makes the approved route easier to use. Give teams a clear list of tools, safe use cases, data rules, reviewer expectations and examples of work that must stay out of scope. Pair that with monitoring signals such as unusual browser extensions, unexplained document metadata, unapproved accounts, staff feedback and recurring quality issues.

Create a governance route for AI incidents

What the incident record should contain

A useful record should be brief enough that busy teams will complete it and detailed enough to support audit, learning and accountability. Include the date, tool, workflow, client or internal area, data category, nature of the failure, reviewer, decision, remediation, disclosure assessment and control change.

For regulated or high-risk work, keep the source output and the corrected version where appropriate. If that creates confidentiality or retention issues, record where the evidence is held and who can access it. The aim is not to punish staff for reporting issues. The aim is to make the firm's use of AI reviewable, defensible and improvable.

Conclusion: make failure safe to report and hard to repeat

AI incidents are not only model problems. They are operating model tests. A firm with clear scope rules, human review, data boundaries and evidence trails can contain failures quickly and improve the system. A firm without those controls may not know whether unsafe AI output reached a client file until much later.

Start with a simple rule: if AI touches client work, the firm needs a named owner, a review standard and a record. If a tool fails or is used without authorisation, the response should protect the client, preserve the facts and strengthen the control environment.

Turn AI governance rules into working implementation steps