“Just use the managed vector store” is the default answer, and it’s a reasonable one — until your product’s core value proposition is finding the right memory even when the wording doesn’t match. At that point, the default answer deserves a real test, not an assumption. So before committing our memory architecture to either a self-hosted vector database (Qdrant) or a managed provider’s vector store, we ran both against the same data and measured.
The setup
We built parallel indexes from real user memory data plus a synthetic evaluation set designed to remove keyword overlap entirely — queries that are semantically related to the target memory but share almost no vocabulary with it. That split matters: a keyword-overlap-heavy test set flatters any system with even weak lexical matching, and hides how well a system does on the harder, more realistic case of “the user described this differently than they described it the first time.”
We measured recall@5 (did the right memory show up in the top 5 results) and MRR (how high it ranked when it did), across both the clean synthetic set and messy real prompts pulled from actual usage.
What we found
On pure semantic matching — no keyword overlap, wording genuinely different from the stored memory — the self-hosted vector database won decisively. That’s the case that matters most for a memory product: the user rarely phrases a question about last month’s decision the same way they described the decision itself.
On real, messy prompts with natural keyword overlap, the managed provider’s store had a modest edge, and it took real digging to figure out why. Our first hypothesis — that the managed store was matching on stored context we weren’t indexing — turned out to be wrong once we checked the actual data. The real explanation was structural: server-side chunking and reranking baked into the managed provider’s pipeline, not something we could replicate by tuning our own settings. Chasing it by shrinking or growing our chunk size didn’t help; we’d already found the chunk size that performed best on real data, and going further in either direction made things worse, not better.
We also tested whether combining semantic and keyword-based (sparse) search would close that gap. It didn’t — on our data, blending the two search styles evenly let the weaker keyword signal drag down results that pure semantic search alone had already gotten right.
The tuning parameters that actually mattered
Two decisions moved the numbers more than anything else:
- Embedding model size. A larger embedding model outperformed a smaller one substantially on the hardest, most semantic queries — the gap was large enough that it wasn’t a close call.
- Chunk size interacting with content shape. The best chunk size for tidy, single-topic content was much larger than the best chunk size for long, multi-topic memories — and most real memories look like the latter. We picked the setting that held up on real data, not the one that looked best on the clean synthetic benchmark.
We also validated that compressing the vectors to save memory (INT8 quantization) cost us nothing — top-5 recall was identical compressed and uncompressed, so we shipped the compressed version by default and got a large reduction in memory footprint for free.
Why we still went self-hosted
The managed store’s edge on real prompts was real but small, and it came from infrastructure we don’t control and can’t inspect — which cuts against the entire portability argument this product is built on. Self-hosting traded that small edge for control over cost at scale, verifiable multi-tenant isolation, and the ability to keep tuning — chunk size, embedding model, quantization — as we learn more, instead of being limited to whatever a managed provider exposes as a setting.
The actual lesson
The honest version of this post isn’t “self-hosted vector search is strictly better.” It’s narrower and more useful than that: run the comparison on your own data before picking a default, expect the two approaches to trade wins depending on how clean or messy your real queries are, and don’t assume a single aggregate benchmark number tells you which one will actually serve your users well.
How we isolated those vectors per user is in Building a Per-User Vector Store System. When vector search is not enough on its own, see Vector Search vs. Knowledge Graph for AI Memory.
Start your 14-day Nexus-Catalyst trial if you want that retrieval layer without running it yourself.