Building a Privacy-First Local RAG System (End-to-End)

Most “RAG tutorials” quietly assume your data is fine to ship to a third-party API. For a lot of real work — health records, internal docs, legal material, anything under GDPR — that assumption is the whole problem. Local RAG flips it: the documents, the embeddings, the vector index, and the language model all stay on hardware you control. Nothing leaves the box.

This is the pillar guide I wish I’d had when I built Cosmind, a local-first second brain that turns a messy Markdown vault into a queryable archive. Here I’ll walk the full pipeline end-to-end, with working code and the design decisions that matter in production — not a toy notebook.

What “Private RAG” Actually Means

RAG — Retrieval-Augmented Generation — is simple in shape: retrieve relevant chunks of your data, stuff them into the prompt, let the model answer grounded in those chunks. The private part is an architecture constraint, not a feature you bolt on later:

• No egress. Documents and queries never hit an external endpoint.
• Local embeddings. The model that turns text into vectors runs on your machine.
• Local vector store. The index lives on local disk, not a managed cloud DB.
• Local generation. The LLM that writes the answer is self-hosted.

If any one of those four leaks to a SaaS API, you no longer have a private system — you have a cloud system with extra steps. That is the whole point: control over data residency is a property of the entire pipeline, not of any single component in it.

The Architecture

Four stages, all local:

┌──────────┐   ┌───────────┐   ┌──────────────┐   ┌─────────────┐
│ Documents│──▶│  Chunk +  │──▶│ Vector store │──▶│ Retrieve +  │
│ (vault)  │   │  Embed    │   │  (ChromaDB)  │   │ Generate    │
└──────────┘   └───────────┘   └──────────────┘   └─────────────┘
                    │                                     │
              local embed model                     local LLM (Ollama)

The stack I’ll walk through here is the one behind Cosmind: Python, Ollama for local generation, and ChromaDB as the on-disk vector store. No API keys, no network calls. (Embedding model choice is your call — Cosmind leans on ChromaDB’s built-in embedder; the code below shows the Ollama route as an equally valid local option.)

Step 1 — Chunking

Chunking is where most RAG systems silently lose quality. Too big, and retrieval pulls in irrelevant text that dilutes the answer. Too small, and you shred the context a passage needs to make sense.

A good default for prose is ~500–800 tokens per chunk with ~10–15% overlap, split on natural boundaries (paragraphs, headings) rather than blindly every N characters:

def chunk_text(text: str, target_chars: int = 2400, overlap: int = 300) -> list[str]:
    paragraphs = [p.strip() for p in text.split("\n\n") if p.strip()]
    chunks, current = [], ""
    for para in paragraphs:
        if len(current) + len(para) > target_chars and current:
            chunks.append(current)
            # carry the tail forward so context isn't cut mid-thought
            current = current[-overlap:] + "\n\n" + para
        else:
            current = f"{current}\n\n{para}" if current else para
    if current:
        chunks.append(current)
    return chunks

Splitting on \n\n keeps paragraphs intact; the overlap tail means a sentence that straddles a boundary still appears, whole, in at least one chunk. Chunking strategy deserves its own deep-dive — it’s the single highest-leverage knob in the pipeline.

Step 2 — Local Embeddings

Embeddings turn each chunk into a vector. Run the embedding model locally with Ollama — nomic-embed-text is a strong, small default:

import ollama

def embed(texts: list[str]) -> list[list[float]]:
    out = ollama.embed(model="nomic-embed-text", input=texts)
    return out["embeddings"]

The choice of embedding model — dimension, language coverage, domain — directly caps your retrieval ceiling. A 768-dim general model is fine for most prose; specialized domains (clinical, legal) reward a domain-tuned model more than any amount of prompt engineering downstream.

Step 3 — The Local Vector Store

ChromaDB persists to local disk and needs no server. Create a collection, add the chunks with their embeddings:

import chromadb

client = chromadb.PersistentClient(path="./vault.chroma")
collection = client.get_or_create_collection(
    name="vault",
    metadata={"hnsw:space": "cosine"},
)

def index_document(doc_id: str, text: str) -> None:
    chunks = chunk_text(text)
    vectors = embed(chunks)
    collection.add(
        ids=[f"{doc_id}::{i}" for i in range(len(chunks))],
        documents=chunks,
        embeddings=vectors,
        metadatas=[{"doc_id": doc_id, "chunk": i} for i in range(len(chunks))],
    )

Cosine distance is the right default for text embeddings. The metadatas matter: keeping doc_id lets you cite which document an answer came from — the difference between a trustworthy assistant and a confident liar. ChromaDB vs pgvector vs Qdrant is a real decision at scale; for a local single-user system, Chroma’s zero-ops disk persistence wins.

Step 4 — Retrieve and Generate

Embed the query the same way, pull the top-k chunks, and ground the model on them. The prompt must do one job above all: answer only from the context, and say so when the context is silent.

def answer(question: str, k: int = 5) -> str:
    q_vec = embed([question])[0]
    hits = collection.query(query_embeddings=[q_vec], n_results=k)
    context = "\n\n---\n\n".join(hits["documents"][0])

    prompt = f"""Answer the question using ONLY the context below.
If the answer isn't in the context, say "I don't have that in my notes."
Cite the source chunks you used.

Context:
{context}

Question: {question}"""

    resp = ollama.chat(
        model="llama3.1:8b",
        messages=[{"role": "user", "content": prompt}],
    )
    return resp["message"]["content"]

That instruction — only from the context — plus a real “I don’t know” escape hatch is the cheapest, most effective hallucination control there is. No amount of model size substitutes for grounding.

The Tradeoffs Nobody Tells You

Running local isn’t free; it’s a different bill:

• Latency. An 8B model on a laptop GPU answers in seconds, not milliseconds. For interactive use it’s fine; for high-throughput, you’ll want a real GPU and batching.
• Quality ceiling. A local 8B model is not GPT-class. For RAG this matters less than people fear — the retrieved context does the heavy lifting, and the model mostly has to summarize faithfully.
• Ops shift. You trade per-token API cost for hardware + maintenance. Past a surprisingly low query volume, self-hosting is cheaper — and the privacy guarantee is categorical, not contractual.

My honest take after shipping this: for most private-data use cases, a small local model with good retrieval beats a frontier model with mediocre retrieval. Spend your effort on chunking and embeddings, not on chasing the biggest LLM.

FAQ

Is local RAG as accurate as cloud RAG? Retrieval quality dominates answer quality far more than model size. With solid chunking and embeddings, a local 8B model produces grounded answers comparable to cloud setups for most document-QA tasks. The gap shows up in open-ended reasoning, not in “find and summarize.”

What hardware do I need to run a local RAG system? A machine with 16GB+ RAM runs an 8B model on CPU slowly but functionally; a consumer GPU (8GB+ VRAM) makes it interactive. Embeddings are cheap and run fine on CPU.

Which vector database is best for local RAG? For single-user, on-disk, zero-ops setups, ChromaDB is the pragmatic default. If you need SQL alongside vectors, pgvector; for larger-scale filtered search, Qdrant.

Does local RAG make my system GDPR-compliant by itself? It removes the biggest risk — third-party data egress — but compliance also depends on access control, retention, and audit. No-egress architecture is a necessary foundation, not the whole story.

Where to Go Next

This is the pillar. The deep-dives that hang off it — chunking strategies, choosing embedding models, hybrid search (BM25 + vectors), cutting hallucinations, and evaluating RAG with golden sets — each take one of these four stages and go all the way down. See more under RAG and software architecture, and the real implementation in Cosmind.

If you’re building private AI for regulated data — health, legal, public sector — this architecture is the baseline I’d start from. Let’s talk if that’s the kind of system you need.