In 2024, a customer asked a Canadian airline’s support chatbot whether he could apply for a bereavement fare after his flight instead of before it. The chatbot said yes, within 90 days. That was false — the airline’s actual policy required the request before travel — but the customer followed the instruction, bought the full-price ticket, and later asked for the fare difference back. The airline refused. The case went to a tribunal, and the airline’s defense was remarkable: the chatbot, it argued, was “a separate legal entity” responsible for its own answers. The tribunal rejected that in a few lines — anything published on a company’s own website, including whatever its chatbot says, is the company’s responsibility — and ordered compensation.
The temptation is to read this as a technology failure. It wasn’t. The technology worked exactly as designed: a language model generated the statistically most plausible answer to the question, with no mechanism stopping it from inventing a policy that doesn’t exist. The real failure happened earlier — nobody designed what should happen when the chatbot hit a question outside what it could answer with certainty.
Why a model doesn’t know when it’s wrong
A language model doesn’t check a company policy database before answering. It predicts the statistically most likely sequence of words, based on patterns in the text it was trained on. For common questions, that almost always produces a correct answer, because the correct answer is also the most frequent one in the data. The problem shows up exactly at the exception — a bereavement fare requested after the deadline, a customer whose case doesn’t fit any standard script — because there’s no dominant pattern in the training data for the model to repeat. It answers anyway, with the same fluency and the same confidence as a correct response. There’s no internal signal in the model’s architecture equivalent to “I’m not sure about this” that fires before it generates the text.
That trait — constant fluency, constant confidence, regardless of whether it’s right or wrong — is what makes a model’s hallucination more dangerous than a human employee’s mistake. An unsure agent hesitates, says they’ll confirm, escalates the question. A model doesn’t hesitate: it completes the sentence with the same ease whether it’s correct or inventing a policy that never existed.
AI executes, the human answers for it
That’s the distinction that should guide any customer-facing, financial, or operational process that incorporates AI: the AI executes the task within what’s already mapped and validated; the human answers for the exception, the ambiguous case, and the reputational risk the model has no way to assess. This isn’t a compliance checkbox to satisfy legal — it’s product design. The point where a human takes over can’t be something the company figures out after the problem already happened; it needs to be designed on day one, alongside the main flow, not bolted on as a contingency appendix.
As João Paulo Batistella, an innovation executive with a track record spanning Ericsson, Telefônica/Vivo, and five years as CEO of EISA, argues, every customer-facing process built on AI needs to anticipate, from the design stage, the moment a human takes over — AI needs oversight, and failure cases must be designed before the system goes live, not discovered after a customer sues the company. In urgent healthcare support, for instance, a chatbot that doesn’t know how to escalate to a human isn’t just inefficient — it’s a safety failure. The same logic applies, with lower stakes but the same principle, to any process carrying high reputational risk: a quarterly report to shareholders drafted by AI without human review carries the same kind of exposure as the airline’s chatbot — a fluent, confident, possibly wrong answer, going out under the company’s name with nobody checking it first.
Three questions to find where the panic button goes
Not every AI-driven process needs the same level of human oversight. A product recommendation engine can be wrong without much consequence; a chatbot promising a refund, or a report going to the board, is wrong with real consequences. Three questions help separate the two before putting any process into production:
1. If the model answers wrong, is the decision reversible? A bad product recommendation costs a lost click. A wrong refund promise or a wrong cancellation-policy answer has already gone out as a commitment from the company to the customer — reversing it takes negotiation, or a tribunal, as in the airline’s case.
2. Is this type of question frequent, or is it the exception the model has never seen? Models are good at the center of the distribution — the question every customer asks — and fragile at the edges, exactly where the bereavement case, the customer with an unusual situation, or the rare contract clause shows up. Processes that regularly deal with edge cases — legal, healthcare, financial — need human oversight by default, not as an exception.
3. Who actually takes over when the model hits its limit? “Escalate to a human” isn’t an answer if there’s no real person, with real time, genuinely on call, to take over at that moment. A panic button that only exists in the project’s flowchart, and not on anyone’s actual work schedule, isn’t a panic button — it’s a statement of intent.
If the answers point to high risk, frequent exceptions, and no human genuinely on call, the process isn’t ready to run on AI alone — it’s ready to run with AI assisting a human, not replacing one.
The panic button is expensive. The process without one is more expensive
Designing the human escalation point has a real cost — training whoever takes over, defining the criteria for when they step in, keeping people available for cases that, on most days, won’t happen. It’s tempting to cut that cost precisely because it looks redundant while everything is working. The problem only shows up on the day it isn’t — and on that day, the cost of not having designed the exception is always higher than the cost of keeping the structure ready. The airline saved the cost of a clear escalation process for months; it paid the price in a tribunal room, and paid again in reputation, at a scale no settlement covers.
Follow Eleva Tecnologia for more analysis on technology applied to business: follow @ElevaTechno on X or @elevatechnologies on Instagram, or learn more about the group at elevatec.net/about.
