Every AI memory demo looks good when the test is:
Remember that my favorite color is green.
Then, ten seconds later:
What is my favorite color?
That proves the system can write and read one obvious fact. It does not prove that it can retrieve the right decision six months later, separate two projects, reject another user’s data, resolve contradictions, or delete every derived copy.
Use these ten questions to evaluate an AI memory product—or an architecture you are building yourself.
1. What problem is the memory solving?
Clarify whether the product provides:
- Personalization inside one chat product
- Search across past conversations
- Project-specific continuity
- Cross-tool shared memory
- Team or organization knowledge
- An application memory API
These are related but different categories. A native assistant feature may be ideal for personalization while remaining unsuitable as a shared memory layer for Cursor and Claude Code.
Ask for a concrete workflow, not a list of storage technologies.
2. Which clients can use the same memory?
“Works with AI” is too vague.
Verify each client and connection method:
- Is the integration public now or merely planned?
- Does it support remote MCP, a local plugin, an API, or copy-paste?
- Is recall automatic or must the user request it?
- Can the client both read and write?
- Does the same identity carry across clients?
Test with two real clients. Store a decision in one, start a fresh session in the other, and retrieve it.
The distinction between native and shared memory is covered in ChatGPT Memory vs. Claude Memory vs. a Shared MCP Memory Layer.
3. How does retrieval work?
Vector similarity is useful but incomplete.
Ask whether retrieval considers:
- Exact keywords and entities
- Project or workspace
- User and team visibility
- Recency
- Superseded decisions
- Memory type
- Source authority
- Duplicate results
- Full detail after summary retrieval
Then test several query types:
- Same meaning, different wording
- Exact URL or person name
- Old decision versus its replacement
- Two projects with similar terminology
- No relevant memory
Read How AI Memory Retrieval Fails for the common failure modes.
4. What gets stored—and who decides?
A system that saves every turn will accumulate noise and sensitive content. A system that saves only explicit commands may require too much ritual.
Evaluate:
- Manual versus automatic capture
- Filters for meaningful decisions and outcomes
- Incognito or do-not-store controls
- Treatment of secrets and sensitive data
- Whether users can preview or edit a memory
- Whether speculation is labeled differently from verified facts
Ask the provider to describe both false positives and false negatives. Memory selection is a product decision, not merely an ingestion pipeline.
5. Can memories be corrected, superseded, and forgotten?
Facts change. The system should not return every historical value as equally current.
Look for:
- Editing or correction
- Supersession links
- Effective dates
- Expiry or review dates
- Archive versus active recall
- Deletion of individual items
- Full-account deletion
The ability to forget is part of memory quality, not just compliance. See What Should an AI Remember—and What Should It Forget?.
6. How is each user and tenant isolated?
For remote or team systems, tenant isolation is non-negotiable.
Ask whether trusted authenticated identity scopes every path:
- Primary database
- Vector search
- Knowledge graph
- Detail records
- Caches
- Exports
- Logs and analytics
Do not accept “we use one database per customer” as a complete answer without understanding the retrieval filters and authorization checks.
Test negative controls. An account must not retrieve a seeded memory owned by another account, even when the query matches exactly.
7. What security model protects the MCP connection?
For a remote MCP server, check:
- HTTPS
- OAuth or equivalent authentication
- Token validation and expiry
- Operation-level authorization
- Tool permissions
- Approval for consequential writes
- Rate limits
- Prompt-injection defenses
- Safe logging defaults
An MCP connection gives a model access to capabilities. It does not make the implementation secure automatically.
Use Is MCP Secure? OAuth, Tenant Isolation, and Data Controls Explained as a deeper checklist.
8. Can you export and move the memory?
Memory compounds in value. That makes lock-in expensive.
Ask what an export contains:
- Full memory text
- Summaries
- Timestamps
- Project and source metadata
- Visibility and ownership
- Entity relationships
- Links to artifacts
- Supersession history
A JSON dump of opaque IDs may satisfy the word “export” without being practically portable.
Also ask whether the connection uses an open standard. MCP can improve client portability, while structured export improves data portability. You need both dimensions.
9. How does team memory work?
Adding multiple users is not the same as designing shared context.
Evaluate:
- Private, team, and organization visibility
- Invitations and membership lifecycle
- Ownership when someone leaves
- Role-based administration
- Promotion of a private memory to shared scope
- Review and correction of team facts
- Retrieval weighting across visibility levels
- Audit history
Ask who is allowed to establish organization-wide truth. A team memory system needs governance as well as search.
10. What happens when the system is wrong or unavailable?
Memory will sometimes miss relevant context or retrieve an incorrect item.
The product should fail transparently:
- Indicate when recall failed
- Show enough provenance to inspect the result
- Surface contradictions
- Let users correct the record
- Continue without memory when the service is unavailable
- Avoid presenting absence of results as proof that no decision exists
Operationally, ask about backups, recovery, incident communication, and graceful degradation.
A 30-minute evaluation script
Create a small test corpus:
- Store a stable preference.
- Store two similar decisions in different projects.
- Store an old decision and a newer replacement.
- Store an episode containing an exact URL or person name.
- Create a negative-control memory in another test account.
Then test:
- Paraphrased conceptual recall
- Exact entity lookup
- Project disambiguation
- Current-versus-superseded ranking
- Cross-client retrieval
- Tenant isolation
- Correction
- Individual deletion
- Export readability
- Behavior when the server is disconnected
Record both relevance and latency. A correct result that arrives after the response is not useful in an interactive workflow.
Evidence to request
Strong providers should be able to show:
- Current connection documentation
- Tool or API reference
- Security and privacy explanation
- Export example
- Deletion behavior
- Clear product limitations
- Status and support channels
Be cautious when future integrations are described as current, retrieval quality is demonstrated only with immediate recall, or privacy claims do not cover embeddings and logs.
Buy the behavior, not the architecture diagram
Vector databases, knowledge graphs, embeddings, and MCP are implementation choices. The product succeeds when it returns the right context, protects ownership, admits uncertainty, and lets users leave with their data.
Evaluate the workflow you actually have. Store a real decision in one session and see whether the right client, user, and project can retrieve it later—without seeing anything they should not.
Test persistent memory with real work
Start your 14-day Nexus-Catalyst trial to evaluate shared AI memory across supported clients using your own retrieval and portability checklist.