Making RAG Production-Ready: Six Pitfalls We Hit

1. Chunking that ignores structure

Fixed-size character chunking tears tables and clauses apart: retrieval finds fragments that cannot answer the question. Chunk by document structure — sections, clauses, tables — and retain the heading path as context.

2. Ignoring permissions

Knowledge bases usually mix classification levels. Retrieval must filter by user role or you get unauthorised disclosure. Tag permissions at ingest, not by filtering results afterwards.

3. No refusal mechanism

Models tend to produce a plausible-sounding answer. Set a similarity threshold; below it, state clearly that no source was found and offer escalation to a human. Better to decline than to invent.

4. Untraceable answers

Enterprise users need to verify sources. Answers must cite the original passage with a link — this is the precondition for trust.

5. No evaluation set

Without one you cannot tell whether a change helped. Build 100-300 covering typical questions before launch, then regress after every change, tracking accuracy and refusal rate.

6. Over-broad tool permissions

Actions touching data modification, money or permissions must keep human confirmation. Grant agent autonomy progressively, graded by risk.