Almost every generative AI project in a mid-sized company today starts with the same promise: “we’ll connect the model to our documents and it will answer based on what is ours.” The technique behind it has a name, RAG (retrieval-augmented generation), and it works. But it has a limit that rarely shows up in the demo: RAG only finds what is written down. A rule that lives only in someone’s head stays invisible, and the AI answers anyway.
How RAG works under the hood
A language model knows nothing about your company. RAG fixes that in three steps, before every answer:
- Split. Manuals, policies, contracts and emails are broken into short passages, called chunks, a few hundred words each.
- Index by meaning. Each chunk becomes an embedding, a list of numbers that represents what the text means. Chunks about the same subject sit close together in that space, even when they use different words.
- Retrieve and answer. When someone asks a question, the question also becomes an embedding. The system pulls the closest chunks, usually 5 to 20, and hands them to the model along with the question. The model writes its answer from them.
Notice what the model receives: not the whole knowledge base, only the chunks the search found. If the right answer is not among them, the model has no way of knowing. And it rarely says “I don’t know.” It writes the best answer it can with what it was given.
Finding the right chunk is already hard
The first failure is retrieval. A chunk loses the context of the document it came from. Anthropic uses a simple example: the passage “the company’s revenue grew by 3% over the previous quarter” names neither the company nor the quarter. On its own, it is almost impossible to retrieve with the right question.
In tests published in September 2024, Anthropic measured how often the needed chunk fell outside the top 20 results: 5.7% of searches in the standard setup. By adding a sentence of context to each chunk, combining meaning-based search with keyword search and reranking the results, failures dropped to 1.9%, a 67% reduction (Anthropic, Contextual Retrieval). It is the vendor’s own test, on codebases, fiction and scientific papers, not on business documents. But it makes the point: retrieval improves a lot with engineering.
All of that improvement, though, works on what already exists in the base. No retrieval technique finds a document that was never written.
What isn’t written doesn’t exist for the AI
Picture an auto parts distributor with 60 employees. It sets up a RAG assistant to help customer service answer questions about exchanges and returns. The base holds the returns policy, a PDF from 2023, and the sales manual.
A customer asks whether they can return an electrical part that has already been installed. The PDF says installed parts cannot be returned. The assistant says so, politely, citing the document. The answer is technically right and practically wrong. For two years, the customer service supervisor has accepted the return when the defect shows up within 7 days, because the supplier refunds the distributor in those cases. Everyone at the counter knows it. Nobody wrote it down.
The system did not fail. Retrieval found the most relevant chunk that existed. The model summarized it well. The problem is that the company’s real rule never became text, so it never became an embedding. The assistant did not hallucinate, in the technical sense. It was faithful to an outdated document. From the customer’s point of view, the result is the same.
As João Paulo Batistella, an innovation executive, argues, AI depends on processes formalized as data, not on knowledge that lives in people’s heads. A routine that exists only as habit gives no AI anything to work on. RAG is the most concrete form of that thesis: it turns the question “does the AI know our company?” into a more uncomfortable one, “is our company written down?”. It is the same problem as a business that depends on a single person, now with a new consequence: that dependency starts answering customers.
The four gaps RAG doesn’t close
In practice, almost every wrong answer from a corporate RAG system falls into one of these four gaps. Only the first is an engineering problem:
- Written, but without context. The passage exists, but once split it lost its subject, date or the unit it refers to. This is the gap that techniques like the one in Anthropic’s test reduce.
- Written in versions that contradict each other. The 2021 policy and the 2024 policy are both in the folder. Retrieval doesn’t know which one applies and may bring back the old one if it happens to look more like the question.
- Written, but outdated. The document was true once, and nobody retired it when practice changed. That is the distributor’s case.
- Never written. Exceptions, verbal agreements, the “this is how we do it here.” It is the most common gap in small and mid-sized companies, and no model, bigger or smaller, solves it.
Gaps 2 to 4 are not IT problems. They belong to whoever owns the process. That is why RAG projects that work usually have a content owner in each area, not just a technical lead.
Four questions before connecting AI to your documents
1. Which questions will the assistant answer? List 30 real questions, taken from customer service, sales or HR. For each one, point to the document that holds the answer. Questions without a document show what needs to be written before you buy anything.
2. Which version applies? Each subject should have one current document, with a date and an owner. Everything else leaves the base, even if it stays archived somewhere else.
3. Who reports when practice changes? A rule change that never reaches the base creates, the next day, an assistant that contradicts your own team. Decide who updates it and how fast.
4. What does the assistant do when it finds nothing? Configure it to say it couldn’t find the information and hand over to a person, instead of answering from the closest chunk. And log those questions: they are your list of the next documents to write.
Where RAG shines, and where it doesn’t
None of this is an argument against the technique. RAG works very well where companies already write everything down because they have to: contracts, technical standards, product catalogs, system documentation, internal regulations. There, it saves hours of searching and reduces made-up answers, because the model cites a source instead of relying only on what it memorized in training.
The limit is in living processes, the ones that change every week and get settled at the counter. There, the work of writing comes before the technology. The good news is that this work pays off even without AI: a company that documents its exceptions trains new people faster and depends less on whoever has been there longest.
RAG doesn’t teach the AI about your company. It reads what your company already knows how to say in writing. The rest stays with the people.
For more analysis of how AI works under the hood and what it changes in management, follow @ElevaTechno on X, @elevatechnologies on Instagram or Eleva Technologies on LinkedIn, or learn more about the group at elevatec.net/about.
