AI9 min read
What is enterprise AI architecture?
The difference between a successful pilot and a production AI capability is rarely the model. It is the architecture around it. Here is what that architecture contains and which decisions matter.
AI
How to build an enterprise RAG architecture
RAG is simple to demo and hard to run. This guide walks through the components of a production retrieval-augmented generation architecture and the decisions that separate a reliable system from a chatbot that hallucinates politely.
Published on 11 min readby Carlos Medger
Key takeaways
- RAG quality is decided at ingestion and retrieval, long before the model writes an answer.
- Access-aware retrieval is non-negotiable in an enterprise: index permissions with the content and filter at query time.
- Hybrid retrieval (vector plus keyword) with reranking outperforms pure vector search on real corporate documents.
- Build an evaluation set before tuning anything, and run it on every change to chunking, embeddings or prompts.
What RAG does and what it does not
Retrieval-augmented generation lets a language model answer questions using documents it was never trained on. At query time the system retrieves relevant passages from a corpus, places them in the prompt and asks the model to answer using that context. The model provides language and reasoning; the corpus provides facts.
RAG does not make a model know your business. It makes the model read the right pages at the right time. If retrieval returns the wrong pages, the answer will be fluent and wrong. That is why most of the engineering effort goes into everything before the model call.
The components of a production RAG system
Ingestion and document preparation
Connectors pull content from SharePoint, file shares, wikis, ticketing systems, ERPs and databases. Each document is normalized, cleaned of boilerplate, split into chunks and enriched with metadata: source, owner, date, classification and the permissions that apply to it. Ingestion runs continuously, not once, because the corpus changes daily.
Chunking
Chunk size and boundaries shape retrieval quality. Chunks that respect document structure (sections, tables, headings) outperform fixed-size windows. Keep chunks small enough to be specific and large enough to carry context, and store the parent section so the model can be given more surrounding text when needed.
Embeddings and indexes
Each chunk is embedded into a vector and stored in a vector index (pgvector, Azure AI Search, Pinecone, Chroma, or the vector capabilities of Databricks and Fabric). Keep a keyword index alongside it. Choose the embedding model with your evaluation set, and plan for re-embedding when the model changes.
Retrieval and reranking
At query time, combine vector similarity with keyword search (hybrid retrieval), filter by metadata and permissions, and rerank the candidates with a cross-encoder or an LLM-based reranker. Return a handful of high-precision passages rather than a long list of loosely related ones.
Generation and citations
The prompt instructs the model to answer only from the retrieved context, to cite sources and to say when the context is insufficient. Citations are not decoration: they are what lets users and auditors verify answers.
Evaluation and observability
A set of questions with expected sources and answers, built with business users, is run on every change. Track retrieval precision, answer faithfulness and latency. In production, log queries, retrieved passages and feedback so the corpus and the pipeline can be improved.
Access control: the enterprise difference
A demo indexes one folder that everyone may read. An enterprise corpus contains HR policies, contracts, incident reports and board material with very different permissions. The rule is simple: the system must never surface a passage to a user who could not open the source document.
Implement it by storing permissions as metadata at ingestion, resolving the user identity at query time and filtering candidates before they reach the model. Do not rely on the prompt to enforce security. Re-sync permissions on a schedule, because access changes.
Integration with enterprise systems
RAG over documents answers questions. Combined with system APIs it can also act: look up the order, check the contract status, open the ticket. This is where RAG meets agents, and where the enterprise integration layer becomes part of the AI architecture. Expose the capabilities agents need through managed APIs with authentication, rate limits and audit, and require human approval for consequential actions.
Common failure modes
- Indexing everything: an unfiltered corpus full of drafts and duplicates degrades retrieval. Curate sources and versions.
- No evaluation set: tuning by anecdote. Build the test set first.
- Pure vector search: misses exact identifiers, product codes and names. Use hybrid retrieval.
- Stale index: documents change, embeddings do not. Run ingestion continuously.
- Security in the prompt: instructions are not access control. Filter at retrieval.
- No cost model: long contexts and many calls add up. Cache, limit and measure per use case.
A reference stack
A typical enterprise stack we design: connectors and pipelines on the data platform (Microsoft Fabric or Databricks), chunks and embeddings in pgvector or Azure AI Search with a keyword index, orchestration in LangGraph with MCP tools for system access, models through a gateway on Azure OpenAI or AWS Bedrock, evaluation automated in CI, and observability on the same monitoring stack as the rest of the platform.
The specific products matter less than the layering. Keep retrieval, orchestration and models separable so each can improve or be replaced without rebuilding the others.
Frequently asked questions
Do we need fine-tuning if we use RAG?
Usually not. RAG solves the knowledge problem; fine-tuning changes style or format. Most enterprise assistants reach their target quality with good retrieval and prompting alone.
Which vector database should we use?
Use what fits your platform. pgvector on PostgreSQL is enough for many corpora; Azure AI Search, Pinecone or the vector features of Databricks and Fabric suit larger or more integrated environments. Decide with your evaluation set and your operations team.
How do we measure whether RAG is working?
Retrieval precision on a labeled question set, answer faithfulness to the retrieved context, user feedback in production and the rate of questions the system correctly declines to answer.
Insights
Keep reading
AI8 min read
AI agents vs traditional automation: when to use each
Agents are not a replacement for every workflow engine and RPA bot. They are a new tool for a specific class of work. Knowing which class is the difference between a reliable operation and an expensive experiment.
Want to apply this to your architecture?
Talk to an architect about your systems, your data and what it would take to execute.