Everyone wants retrieval-augmented generation now. But a whole class of organizations can’t take the easy path to get it.
A hospital, a law firm, a bank, a defense contractor — anyone operating under a data-residency regime — cannot ship their documents and their users’ questions off to a hosted model API. “We sent our patient records to a third party so the chatbot could read them” is not a sentence their compliance office will ever sign. For these organizations the interesting engineering constraint isn’t retrieval quality or prompt design. It’s simpler and much harder: the data cannot leave.
That constraint is where I’ve been living, and it’s the reason I’m building CorpusCloudKit.
From the device outward
I’ve spent a while building privacy-first RAG at the smallest possible scale — entirely on-device, on an iPhone. My CorpusKit tools ingest documents, chunk them, embed them with a local Core ML model, and answer questions with retrieval plus on-device inference. Nothing ever leaves the phone. That’s a strong privacy story precisely because the boundary is so tight: there’s nowhere for the data to go.
CorpusCloudKit is what happens when you lift that same architecture up exactly one tier — from “stays on the device” to “stays inside the customer’s own environment,” their VPC or their on-prem network. Same pipeline: ingest, chunk, embed, retrieve, generate. Same .corpus bundle format, same embedding model (all-MiniLM-L6-v2), so a corpus built on the device loads straight into the service with no re-embedding. The difference is that now it runs in a container behind a REST API instead of inside an app.
And here’s the thing I keep coming back to while designing it: in a private RAG system, the privacy boundary isn’t a feature you add. It’s the product. The vector search, the chunking, the choice of model — that’s all comparatively commodity now. What’s genuinely hard, and genuinely valuable, is being able to say “your data never leaves this boundary” and mean it in a way someone else can verify.
Making the boundary mechanical
Most “private” or “secure” RAG pitches assert the property. Your data stays with you. Trust us. But assertion isn’t architecture, and “trust us” is exactly the thing a compliance reviewer is paid not to do.
So the design goal I set was: make the boundary something you can inspect, not something you take on faith.
Concretely, the LLM backend is a pluggable interface, and every backend has to declare one boolean: is_local. Ollama or vLLM running in the same container? Local — no data leaves. A hosted API like a cloud model provider? Not local — data crosses the boundary. The service records which backend is configured and reports it at /health as one of two states:
boundary: "contained"— the configured backend is local; nothing can leave.boundary: "egress-configured"— an outbound path exists, deliberately turned on by the operator.
The privacy claim becomes a status you can curl, not a paragraph in a sales deck. In a regulated deployment that distinction is the whole ballgame: it’s the difference between “we believe this is private” and “here, watch me show the auditor it’s private.” The default configuration — local embeddings, local LLM — makes zero outbound calls, and the service says so about itself.
I like this because it turns a value into a mechanism. The privacy isn’t a promise sitting in documentation; it’s a property the running system reports on and enforces.
The rest of the architecture, briefly
Around that core idea, the shape is deliberately boring:
- Ingestion — load a signed
.corpusbundle, or ingest raw documents through the same chunk-and-embed pipeline. - Storage —
pgvectoron Postgres. One dependency, available as a managed service on the major clouds, and it keeps chunks, embeddings, and metadata in one auditable place. When you’re trying to prove where data lives, “it’s all in this one database” is a feature. - Retrieval — semantic search over the embeddings, with an optional expansion step for bridging the gap between the words a user types and the words the source actually uses.
- Generation — assemble the retrieved context, call the configured local backend, and bind citations back to source chunks so every answer is traceable to where it came from.
- The service — a containerized Python/FastAPI app with a REST API, webhooks, structured logging, and a metrics endpoint.
docker compose upbrings the whole thing — app, database, and local model — up on one host.
Boring is the point. The interesting decisions live at the edges.
Two decisions that show the reasoning
A tutorial follows the happy path. A real system is mostly the judgment calls, so here are two.
When not to generate. A RAG system doesn’t have to answer. If retrieval comes back weak — nothing in the corpus is actually relevant to the question — then generating on top of that thin context is precisely how you manufacture a confident, well-written, wrong answer. So the service can return the retrieved passages and decline to generate, handing back grounded material instead of an invented paragraph. I’ve come to think knowing when the LLM is the wrong tool is the actual skill, not a caveat to it.
Deferring the reranker. Vector search is a two-stage problem. The first stage — the bi-encoder embedding search I described above — is fast because the document vectors are precomputed, but it’s only so-so at ordering results. The genuinely best passage might come back ranked eighth. A second stage, a reranker, fixes that: a cross-encoder looks at the query and each candidate together and rescores them for relevance, pulling the right passage up to the top. It’s more accurate but slower, because nothing can be precomputed — it runs fresh at query time.
I’m deferring it from the first version. Not because it doesn’t matter, but because the first version’s job is to prove the end-to-end pattern, and first-stage retrieval plus the expansion step is enough to demonstrate that. So I’m shipping a no-op reranker behind a clean interface — a seam — and a real local cross-encoder drops in later with no other changes.
There’s a wrinkle that I find satisfying, because it cascades straight from the core principle: the convenient reranker would be a hosted reranking API. But calling one would ship your candidate passages out over the network — which breaks the containment guarantee that is the entire point of the system. So any reranker here has to be local, same as the LLM. The privacy principle isn’t a layer on top; it constrains every component underneath it.
What I left out on purpose
The first version is a single-tenant reference service. Deliberately not in it: multi-tenancy, role-based access control, SSO, ingestion connectors for a dozen SaaS apps, high-availability autoscaling. Those are real, and they’re on a roadmap, but none of them is the architecture — they’re what you add after the architecture is proven. There’s a well-funded field of private-RAG platforms already; I’m not trying to out-build them. I’m building the smallest thing that demonstrates the pattern cleanly, and being able to say precisely what I left out and why is, I think, more of an engineering signal than a sprawling half-finished platform would be.
Where this is
To be clear about status: this is the architecture and a v0 build in progress, not a finished product. I’m building it in the open. The reason I’m writing about it before it’s done is that the design decisions — the boundary made mechanical, the local-only constraint cascading through every component, the discipline about what to leave out — are the interesting part, and they’re settled. The code is the execution of choices already made.
If you work somewhere that wants RAG but can’t send its data to a third party — regulated, air-gapped, privacy-first by mandate or by conviction — this is the shape of a system built for you, where “the data never leaves” is something the service will tell you about itself.
More as it comes together.
I build privacy-first iOS/macOS apps and private AI systems. Portfolio at robroy.online, code at github.com/robert-e-roy.