Semantic memory search sounds simple in a demo: embed the text, store the vector, query by similarity. It stops being simple the moment you have more than one user, because the failure mode of getting isolation wrong isn’t a bug report — it’s one user’s private context showing up in another user’s search results.
Why “one big index” doesn’t work
The naive approach is a single vector index for everyone, with a user_id field you filter on at query time. It’s the fastest thing to build and the easiest thing to get subtly wrong. Filter logic that’s correct in the common path and wrong in an edge case — a missing filter on one query path, a batch job that forgot the tenant boundary — turns into a cross-tenant data leak, not a degraded search result. For a memory system whose entire premise is “this is your private context,” that’s not an acceptable risk surface no matter how rare the edge case.
We designed around isolation as a structural property, not a runtime check: each user’s vectors are addressable and queryable in a way that makes “accidentally search across users” require deliberately bypassing the isolation boundary, not just forgetting a WHERE clause.
What we actually evaluated
Before committing to an architecture, we ran a real comparison rather than assuming the answer: a self-hosted vector database (Qdrant) against a managed provider’s vector store, across both clean semantic queries and messy real-world prompts pulled from actual usage. The short version — covered in more depth in a companion post — is that the self-hosted route won on the axis that matters most for memory recall (finding the right memory when the wording doesn’t match), and gave us levers a managed store doesn’t: quantization to cut memory footprint, and multitenancy controls we could verify ourselves instead of trusting a vendor’s isolation guarantees.
Design choices that mattered more than the vector math
- Embedding model and chunk size aren’t independent decisions. Larger embedding models win decisively on pure semantic matching; chunk size interacts with how memories are actually shaped in practice — a single short note behaves differently than a long, multi-topic memory once it’s split into chunks. We picked a chunk size validated against real memory shapes, not synthetic benchmarks alone.
- Precision beats recall by default, then you tune. A memory system that surfaces a barely-related result alongside the right one is more annoying than one that occasionally misses — false positives erode trust faster than false negatives. We biased toward precision and treated recall improvements as something to add carefully, with measurement, not by loosening thresholds.
- Consent and scope have to gate anything cross-user. Any feature that could plausibly touch more than one person’s data — team memory sharing, aggregate product-improvement signals — sits behind an explicit consent and scope check before it ships, not after. Building the isolation boundary right the first time is what makes those features possible to add later without redesigning storage.
The lesson underneath the implementation
Per-user vector storage isn’t a scaling problem you solve once traffic grows. It’s a trust problem you have to get right before the first real user’s data goes in, because the cost of fixing an isolation bug after the fact isn’t a patch release — it’s an incident. We built it that way from the start rather than retrofitting it once memory search worked well enough for a demo.
The database comparison behind that choice is in Choosing a Vector Database for AI Memory: What We Learned Evaluating Qdrant. Tenant isolation at the MCP boundary is in Is MCP Secure? OAuth, Tenant Isolation, and Data Controls Explained.
Start your 14-day Nexus-Catalyst trial to use a memory store isolated to your account.