Building a Production-Ready RAG System for Enterprise Documents

Consider an organization with fifteen years of internal documentation spread across SharePoint, Confluence, a document management system, and a few thousand PDFs that nobody has fully indexed since 2019. Someone asks a chatbot “what’s our current policy on vendor data retention for EU customers,” and the honest answer requires reconciling three documents, one of which was superseded eighteen months ago. This is the actual problem retrieval-augmented generation is supposed to solve, and it’s considerably harder than the “upload a PDF, ask it questions” demos suggest once you add real document mess, real access control, and a real cost of being wrong.

Why naive RAG breaks in the enterprise

The reference architecture everyone starts with — fixed-size chunks, embed, store vectors, retrieve top-k by cosine similarity, stuff into a prompt — works fine for a single clean PDF and falls apart under three enterprise-specific pressures:

1. Document heterogeneity — tables inside PDFs, scanned contracts or insurance claim forms with no text layer, decks used as documentation, Confluence pages with nested macros, Word docs with tracked changes embedded in the XML.
2. Access control — a system that returns the right answer from a document the requesting user isn’t authorized to see isn’t helpful; it’s a data breach with a friendly UI.
3. Cost of hallucination — a wrong answer about a retention policy or a manufacturing tolerance isn’t a UX issue, it’s a compliance or safety incident. “Confidently wrong” is worse than “slow but correct,” which inverts intuitions teams bring from consumer chatbot work.

Everything below is organized around fixing these three, not technology choices for their own sake.

Ingestion and parsing: where most RAG projects actually fail

Teams spend disproportionate design attention on the vector database and little on ingestion, then wonder why retrieval quality is poor — an excellent reranker cannot recover information that parsing destroyed.

Real enterprise document sets typically require:

• Layout-aware PDF extraction, not naive extraction — a PDF from a page-layout tool often has text runs in an order unrelated to reading order (multi-column layouts, interleaved headers/footers); tools like Azure AI Document Intelligence or Unstructured.io that reconstruct layout beat pdftotext meaningfully.
• Tables extracted as structured data (Markdown or JSON) and kept as their own chunk, not flattened into comma-separated numbers that lose the row/column relationship that gave the table meaning.
• OCR with a quality gate — output on a faxed 2011 contract or a scanned insurance claim form can be bad enough to be actively harmful; track a confidence score per page and flag or exclude low-confidence chunks rather than silently indexing them.
• Connectors that preserve metadata, not just content — permissions, last-modified date, author, source library — captured at parse time rather than discarded, since it’s what makes access control and staleness handling possible later.

A pragmatic ingestion pipeline for a mixed enterprise document set typically looks like:

flowchart TD
    A[Source connectors\nSharePoint / Confluence / DMS] --> B{Document type}
    B -->|Native PDF, DOCX| C[Layout-aware extraction\nAzure Document Intelligence / Unstructured]
    B -->|Scanned image PDF| D[OCR + confidence scoring]
    B -->|HTML / Confluence| E[HTML-to-structured-text\nstrip macros, keep headings]
    C --> F[Structure reconstruction\nheadings, tables, sections]
    D --> F
    E --> F
    F --> G[Chunking\nsection-aware, table-preserving]
    G --> H[Metadata enrichment\nACLs, doc version, dates, source URL]
    H --> I[Embedding generation]
    I --> J[(Vector store +\nkeyword index)]
    H -.->|low OCR confidence\nor parse failure| K[Human review queue]

The review queue isn’t optional polish — an ingestion pipeline HzMinds would typically design includes an explicit path for documents that fail to parse cleanly, because silently indexing garbage is worse than flagging it and moving on.

Chunking: why fixed-size windows fail on structured documents

Fixed 512-token windows with overlap are the tutorial default, and they degrade badly on structured enterprise content — cutting mid-table, mid-procedure-step, mid-clause — leaving a chunk that’s no longer one coherent idea and is missing the context that gave it meaning.

Structure-aware chunking holds up better:

• Split on document structure recovered during parsing (headings, sections, list boundaries), not raw character counts.
• Keep tables atomic; if one exceeds the embedding model’s context, chunk by row groups and repeat the header row — a stripped header is one of the more common ways a correct retrieval turns into a wrong answer on a benefits table, a rate schedule, or a parts catalog.
• Treat chunk size as an upper bound, not a target: “one section, or a fragment of one,” not “exactly 512 tokens regardless of content.”
• Attach parent context — the section heading, ideally a one-line summary — prepended before embedding or carried as metadata for generation time.

Every chunk needs metadata beyond its text, since that’s what makes filtering, citation, and freshness handling possible downstream:

Embeddings and vector search: picking a stack that fits the actual scale

Embedding model choice matters less than teams assume past a baseline quality bar; the real risk is a license, hosting, or context-window mismatch, or a model that’s expensive to re-embed later. On Azure OpenAI already, a current-generation text-embedding-3-large-class model is a reasonable default mainly because it stays inside the stack’s compliance boundary — that matters more to procurement than a marginal benchmark difference. Track the model version in metadata: upgrading means re-embedding the whole corpus, since old and new embeddings don’t share a vector space, so budget re-indexing as a recurring job.

Vector store choice should follow existing infrastructure and scale, not whichever tool is trending:

For most mid-size deployments — a financial services back office or a healthcare document repository in the hundreds-of-thousands-to-low-millions-of-chunks range — pgvector or Azure AI Search cover it without a new database technology to operate.

Cost and effort trade against each other fairly predictably, worth budgeting even in hedged terms: a self-hosted option (pgvector, Milvus/Qdrant OSS) typically costs less in infrastructure spend but adds real operational work — for a corpus this size, a handful of engineer-weeks for initial tuning and upgrade setup, plus ongoing part-time attention, is a more realistic assumption than a one-time cost. A managed option (Azure AI Search, Pinecone, Weaviate Cloud) usually costs more per month at comparable scale but removes most of that ops burden — often the better trade without spare platform capacity. Treat these as illustrative bands, not benchmarked figures; a short scoping exercise against your own corpus beats reasoning from a vendor’s list price.

Hybrid search and reranking: why retrieval width is a tuning decision

Pure vector similarity retrieves semantically related content, which is exactly the wrong behavior when a user searches for an exact term — a policy number, an error code, a SKU, a claim number — needing an exact match, not a “similar meaning” one; embeddings are also weaker on rare tokens and proper nouns than on general semantic content. The fix is hybrid search: run keyword search (BM25, or full-text search in Postgres/Azure AI Search) alongside vector search and combine the results, typically with reciprocal rank fusion, rather than relying on either alone.

Both retrieval modes optimize for recall over precision, so hybrid search still returns a noisy candidate list — which is what a reranking step (a cross-encoder like Cohere Rerank or a self-hosted bge-reranker, scoring each candidate against the query) is for. The width of that candidate set is worth tuning deliberately, not copying from a blog post. Too narrow — top 10 — and the correct chunk sometimes never reaches the reranker at all: it can only re-order what it’s given, so a recall miss upstream is unrecoverable no matter how good the reranker is. Too wide — top 100 or more — and cost and latency climb roughly linearly, since a cross-encoder scores candidates individually rather than in one batched comparison; past a certain width the recall gain flattens while p95 latency and per-query cost keep rising. Retrieving the top 25–50 candidates and reranking down to the 5–8 chunks that enter the prompt is a reasonable starting band, but the right numbers for a given corpus come from plotting recall@k against a golden query set at a few widths and picking where the curve stops improving — not a default carried over from someone else’s corpus or benchmark.

Retrieval-time access control: the part that isn’t optional

This is the requirement that separates a working demo from a deployable system. If document-level permissions exist in the source, the pipeline must enforce them at query time, not just at ingestion — permissions change after documents are indexed, and a static one-time filter goes stale within days.

The pattern that works: mirror source-system ACLs into chunk metadata at ingestion, and apply that ACL as a hard pre-filter on the vector search itself, never as a post-retrieval step that discards results afterward. Filtering after retrieval is a subtler bug than it sounds — retrieve top-8, remove the 3 the user isn’t authorized to see, and they silently get 5 instead of 8, with nothing erroring to signal it. In a financial services back office that’s an analyst on a restricted deal team quietly getting a thinner answer with no indication why; in a healthcare document repository, the same bug can mean a clinician doesn’t see guidance they were actually entitled to see.

Two mechanics under that principle pass a demo cleanly and fail only once real permission structure and scale show up:

• Flatten group hierarchy before it hits the filter. Most vector store filter syntaxes support flat equality or $in against a scalar or array field — not graph traversal of nested Active Directory or Entra groups, or “member of a group that’s a member of a group.” Precompute each document’s effective ACL — the fully expanded group IDs that actually grant access — at ingestion, the way a materialized view precomputes a join, rather than resolving hierarchy inside the retrieval-time filter itself, which usually can’t express it and adds per-query latency even where it can.
• Confirm the filter runs before the approximate search, not after it. “Supports metadata filtering” doesn’t always mean pre-filtering — some ANN implementations over-fetch a candidate set and discard non-matching rows afterward instead of restricting the graph traversal itself. Azure AI Search’s security-trimming filters and pgvector’s WHERE clause on an indexed column are genuine pre-filters; other configurations only approximate one. For a narrowly-scoped user — a small deal team, a manufacturing engineer’s specific certification group — that over-fetch-then-discard approach can silently return far fewer than k results even when enough authorized matches exist, because the ANN traversal never considered the filter while choosing which vectors to visit. It’s the same failure as application-level post-filtering, just moved a layer down into the store — worth verifying directly rather than assuming it from the product page.

For permission changes outside the ingestion cycle, don’t rely solely on periodic reindexing — subscribe to permission-change events where possible, or set an explicit staleness SLA (e.g. permissions refresh within 15 minutes) rather than leaving it unstated.

# ACL enforced as a hard pre-filter, applied before ranking — not after.
from dataclasses import dataclass

@dataclass
class RetrievalRequest:
    query: str
    user_group_ids: list[str]   # from the identity provider, never client-supplied

def retrieve(request: RetrievalRequest, vector_store, keyword_index, reranker, top_k=8):
    # Pre-filter: only chunks whose acl_tags intersect the caller's actual
    # effective group membership are eligible candidates at all — applied
    # by the store itself, before the ANN search runs, not after.
    acl_filter = {"acl_tags": {"$in": request.user_group_ids}}

    vector_hits = vector_store.similarity_search(
        query=request.query, filter=acl_filter, k=40,
    )
    keyword_hits = keyword_index.search(
        query=request.query, filter=acl_filter, k=40,
    )

    # reciprocal_rank_fusion(...) merges both ranked lists by position,
    # not raw score — the standard RRF combination, omitted here for brevity.
    fused = reciprocal_rank_fusion(vector_hits, keyword_hits)
    return reranker.rerank(query=request.query, candidates=fused, top_k=top_k)

user_group_ids must come from a trusted source — the identity provider or a backend session, never a value the client can influence — for the same reason any authorization check must be server-enforced. This is the same class of problem discussed in designing a multi-tenant SaaS platform with ASP.NET Core: tenant isolation enforced at the data layer, not trusted to application logic alone, just applied to a vector index instead of a relational database.

Handling conflicting and outdated documents

Enterprise document sets accumulate superseded versions — the 2022 policy and the 2024 policy both still exist, or a manufacturing parts catalog still carries a spec sheet for a part revised eighteen months ago — and every version gets retrieved as semantically relevant to the same query unless something actively deprioritizes the stale one. Realistic mitigations, roughly in order of effort:

• Version and effective-date metadata, with retrieval logic that either filters to the current version when a superseded_by link exists, or boosts recency in ranking.
• Explicit document lifecycle management upstream — flagging or archiving superseded documents in the source system itself, which fixes the problem at the root rather than compensating for it in retrieval.
• Surfacing the conflict rather than hiding it when the system genuinely can’t tell which version is authoritative: an answer that says “two versions of this policy exist — the 2024 version states X, the 2022 version states Y” is more useful and more honest than confidently picking one.

Grounding the answer and mitigating hallucination

Retrieval quality bounds answer quality, but generation still needs explicit constraints — a well-retrieved context doesn’t stop a model from ignoring it:

• Citation requirements in the system prompt, so each claim maps to a specific chunk (inline markers resolved to document, section, page); require answering only from context and saying “I don’t know” rather than filling gaps from parametric knowledge.
• Post-hoc citation verification — a lightweight check (string overlap, or a smaller model call) confirming the cited chunk actually supports the claim attached to it.
• Confidence signaling — if reranker scores are low across the board, say so rather than generating a fluent answer from thin evidence.
• Context window discipline — 20 chunks “to be safe” degrades quality and cost; 5–8 well-reranked ones outperform a large, noisy context, the same discipline retrieval-width tuning above is aiming at.

Evaluation and monitoring in production

RAG quality degrades silently — a document gets reworded, a chunking edge case starts truncating tables, an embedding model gets deprecated — and without deliberate evaluation the first sign is a user complaint. A minimum viable setup:

• A golden query set — 50–200 representative questions with known-correct source chunks, built from real usage once available and subject-matter-expert input before launch.
• Retrieval metrics over time — recall@k and MRR against the golden set, run on every significant pipeline change, including changes to retrieval width.
• Answer-level evaluation — typically LLM-as-judge scoring for faithfulness and relevance, spot-checked against human judgment.
• Production monitoring — log queries, retrieved chunk IDs, and answers (with an appropriate redaction/retention policy); track “I don’t know” rate and user feedback as leading indicators of drift.

Latency deserves explicit monitoring too, for the same reasons discussed in why enterprise APIs get slow at scale: a hybrid search + rerank + generation pipeline has several serial stages, and p95 latency creeps up as corpus and query volume grow without anyone noticing until users complain.

When a full RAG system is overkill

RAG is not the right answer to every “search our documents” problem:

• Small, static, well-organized document sets — a few hundred documents that rarely change, whether an HR policy library or a manufacturing SOP binder — are often served better by faceted search plus a maintained FAQ: lower cost, no hallucination risk, easier to audit.
• A single deterministic source of truth (a current SKU price, an account balance) belongs in a direct database or API lookup, not a probabilistic pipeline that adds latency and uncertainty for a question with one correct, structured answer.
• Low query volume relative to build cost — a handful of queries a week may not justify maintaining ingestion pipelines, evaluation sets, and ACL mirroring; a well-indexed intranet search may serve better.
• Zero-tolerance-for-error decisions — RAG reduces but doesn’t eliminate hallucination risk, so a human-in-the-loop workflow with RAG as a research aid, not an autonomous answer engine, is the more defensible design for certain legal or safety-critical judgments.

Building this well — the ingestion pipeline, hybrid retrieval, access control, and evaluation loop together — is the kind of system HzMinds’ engineering team would typically treat as a genuine engineering project with its own testing and monitoring discipline, not a weekend integration on top of an LLM API.

Key takeaways

• Ingestion quality, not vector database choice, is the biggest lever on RAG answer quality — invest in layout-aware parsing, table handling, and OCR confidence gating first.
• Structure-aware chunking that preserves tables and section context outperforms fixed-size windows on enterprise documents.
• Retrieval width is a tuning decision, not a default: too narrow and the reranker never sees the right chunk; too wide and cost/latency climb for flattening recall gains — tune it against a golden query set.
• Access control must be a hard pre-filter at retrieval time, sourced from a trusted identity provider — confirm it runs before the approximate search, not after, with group hierarchy flattened into an effective ACL first.
• Mitigate hallucination with citation requirements, post-hoc citation verification, explicit “I don’t know” behavior, and disciplined context sizing.
• Treat evaluation as ongoing, not a launch checkbox — golden query sets and production monitoring catch the silent degradation RAG systems are prone to.
• Self-hosted vector infrastructure trades lower direct cost for ongoing engineering effort; managed options trade the reverse — budget both sides, not just the sticker price.
• Not every document-search problem needs RAG — small static corpora, single-source-of-truth lookups, and extremely high-stakes decisions are often better served by simpler or more controlled approaches.