An internal document assistant: the decisions that make or break RAG

“Let’s connect all our documents to an AI and let people ask questions” sounds simple, and the technical setup really is about a week of work. What sinks these projects is not the model — it is the decisions that determine which document answers whose question.

First decision: scope

Indexing everything is the worst possible start. Procedures from 2019, cancelled price lists and three versions of the same handbook make the assistant produce confident, wrong answers. Start with one high-value domain: HR procedures, product technical documentation or after-sales service manuals. An assistant that is right in a narrow domain beats one that is unreliable across a broad one.

Chunking decides quality

Documents are split into chunks and the closest ones are retrieved. Cutting at a fixed character count slices tables and clauses in half, and half the answer disappears. What works in practice:

• Split along the heading hierarchy and prepend parent headings to every chunk.
• Never split tables; if you must, repeat the header row in each part.
• Attach source metadata to every chunk: document, version, date, section.
• Overlap chunks slightly so sentences on the boundary are not lost.

One retrieval method is not enough

Semantic search handles conceptual questions well but is weak on exact matches such as a form name or a product code. Combine semantic search with keyword search and re-rank the results. Enterprise content is full of codes and abbreviations, so this hybrid makes a measurable difference.

Permissions belong inside the index

The assistant must never answer from a document the user cannot see. Filtering after generation is too late — the content already reached the model. Permission metadata must sit on every chunk and retrieval must be restricted at the first step. When salary bands or contracts leak, it is not a bug, it is a breach.

Citations are mandatory

Every answer must show which document and section it came from. Users can verify, and a model forced to cite hallucinates noticeably less. When nothing relevant is found, the correct behaviour is “I could not find this in the documents” rather than a guess. Make that explicit in the system instructions.

Freshness and versions

When a document changes, its old chunks must leave the index — otherwise the assistant keeps defending a cancelled procedure. Show the date of each source in the answer. An assistant whose freshness is unmanaged becomes a tool nobody trusts within months.

How to measure success

Before launch, build a test set of 50–100 real questions with answers written by people who know the subject. For every release measure three things: answer accuracy, retrieval accuracy, and the hallucination rate on questions that should return “I don’t know”. In production, a thumbs button under each answer is the cheapest useful feedback loop you can build.