RAG for company documents: when it helps and how to check it

RAG combines information retrieval with answer generation. Google Cloud documentation describes this approach for supplying external context to a model. It does not guarantee that retrieval finds the right passage or that generation interprets it correctly. This guide proposes assessing those two stages separately.

Imagine a January support manual specifying a 48-hour response, a March update changing it to 24 hours and an unapproved draft proposing 12 hours. The test question is “Which deadline should we communicate?” Before using AI, establish which document has authority, from when and for which service.

Keeping versions does not make all of them current. Include status, effective date, owner and permitted audience. If metadata cannot tell you which document governs, a stronger search engine will not solve the problem. You need a document policy.

Try equivalent questions about response deadline, maximum reply time and support commitment. Record the retrieved passages before drafting an answer. A passing test retrieves the applicable update with its context. Returning only the draft, or cutting away a service exclusion, fails.

Include a question not answered by the documents. Missing evidence is a useful outcome: the system must be able to ask for help. Do not fill your test set only with questions you already know work.

Request a short answer naming the document, section and effective date. Check whether each claim is supported by the retrieved passage. A real citation that does not support the sentence is not evidence. Conflicting sources should be explained or escalated under your document policy.

Create two test profiles with different permissions and repeat a question about a restricted document. The source must not enter an unauthorised user’s context. Hiding its link after generating an answer is too late if the text already contains restricted information.

A small, stable collection may be served by a maintained FAQ. Exact record lookup may be easier to verify with field-based search. Consider RAG when natural-language questions offer a concrete advantage and you can maintain sources, permissions and evaluation. This article proposes a test design, not a completed implementation.