Neuroscience Inspired AI Memory
Arlo Gilbert · July 8, 2026
Your AI forgot something you told it last week. If AI memory worked like a human brain, it would remember.
That is the failure I keep seeing in products that bolt a vector database onto an LLM and call it memory. A task due today and a principle you learned six months ago both sit at roughly the same cosine distance from the query. The system cannot tell urgent from archival, emotional from routine, skill from trivia. It retrieves what looks similar. Humans recall what mattered, what was practiced, what was pinned, and what was reinforced when two ideas fired together.
We did not want that for our LLM powered products. So we built a seven-layer neuroscience inspired memory stack, orchestrated through ZenBrain, stored in LanceDB, embedded with Jina, and wired so chat never waits on memory to answer. If you aren't familiar with LanceDB, I highly recommend checking it out. LanceDB lets you use object storage for vectors so you never need a cluster of Mem0 or Qdrant servers... and you'll definitely never need something like SuperMemory or Honcho, those are great products, but they are more like toys compared to what we use.
This post is the honest version: what we built, why these choices, what it costs, and where it still hurts to debug.
The problem: vector search is an archive, not a mind
Simple RAG is seductive because it is easy to explain. Embed documents. Embed the question. Return the top k matches. Ship.
The metaphor breaks the moment your product needs continuity. Events should not decay like facts. "We talked about GDPR mapping on Tuesday" is not the same kind of thing as "GDPR applicability depends on controller versus processor roles." One is episodic context. One is semantic knowledge. A flat vector index has one knob: similarity.
Skills are worse. "How to run a privacy audit" is procedural. You do not want that memory rewritten every time someone mentions audits in passing. You want it refined when the workflow is demonstrated and reused.
Identity is another category entirely. "I work in data privacy" should not get consolidated away because it has not been retrieved lately. That is core memory, not a ranked search hit.
Relations matter too. Two facts can be boring alone and powerful linked. Vector stores do not natively express supports, contradicts, causes, or requires. You can fake it with prompt stuffing. You cannot learn from co-activation.
Context should shape recall. A creative marketing tip from last week might be semantically close to a technical question today and still be the wrong memory to surface.
I am not dunking on RAG. RAG is a great retrieval pattern. It is a bad memory model when you want software that feels like it knows you over months, not like it reread your wiki.
The solution: seven layers, one coordinator
We split memory by cognitive role and gave each layer its own update rules. ZenBrain coordinates the layers. LanceDB holds the durable rows. A knowledge graph sits alongside with typed epistemic edges.
Here is the stack:
Working memory is what you are holding right now. Small slot budget, fast decay. This is the "in this second" focus buffer.
Short-term / session memory is the live chat transcript window. FIFO, bounded, gone when the session ends. No pretense of permanence.
Episodic memory stores concrete events. "On Tuesday you asked me to map GDPR implications for your workflow." Emotional intensity can preserve episodes longer than bland exchanges.
Semantic memory stores distilled facts with FSRS scheduling. Difficulty, stability, next review date. Facts strengthen when recalled at the right time and fade when they stop earning their place.
Procedural memory stores workflows and skills. Triggers, steps, demonstrations. It does not behave like a fact table row that should be overwritten on every similar mention.
Core memory is pinned identity and role context. Always loaded. Not subject to consolidation games.
Cross-context memory deduplicates entities across domains. Work Alice and personal Alice should converge when they are the same person, not duplicate forever because the embedding clusters looked different.
The graph layer is the bonus people forget to count. Nodes plus edges like supports, contradicts, causes, requires. That is where Hebbian strengthening, Bayesian confidence propagation, and sleep-style replay actually live.
Why seven instead of one table with a type column? Because access patterns differ. A fact that never gets used should decay. A core block should not. A skill should improve through practice. A session buffer should disappear. Treating them equally is how you get memory that feels clever in a demo and incoherent in month three.
Why ZenBrain, LanceDB, and Jina
LanceDB instead of Postgres plus pgvector
We wanted schema, vectors, and columnar storage in one place. A semantic fact in our world carries an embedding, FSRS state, JSON metadata, and sometimes emotional tags. In LanceDB that can live in one row model built on the Lance columnar format. LanceDB's own materials describe embeddings as columns alongside metadata, which matches how we think about memory records rather than "documents."
Postgres plus an extension would have worked. We would have spent more time joining tables and fighting migration drift for a workload that is read-heavy, embedding-heavy, and multi-tenant at scale.
LanceDB also fits our deployment story: local filesystem for dev, object storage backends for production paths, same code. For a SaaS product that has to move data between environments without rewriting retrieval, that matters.
The trade is real. Lance is newer than Postgres. Schema evolution is intentional friction. Breaking changes mean read-old, write-new, atomic rename migrations. I prefer that pain to silent ALTER TABLE drift in a memory subsystem where a bad migration leaks across tenants.
Managed vector databases optimize for one question: given a query embedding, return top k by distance. That is great for search. It is thin for FSRS state, graph edges, consolidation metadata, and per-layer routing unless you bolt on more systems. We already have enough systems.
ZenBrain instead of inventing consolidation from scratch
Eight tables, dozens of columns, FSRS updates, Hebbian edge weighting, sleep replay selection, multi-signal ranking. Building that from zero is two or three engineering years if you want it tested under load, not a weekend hackathon.
ZenBrain gave us a reference orchestration model and algorithms grounded in spaced repetition research and systems that already tried to make LLM memory less dumb. We customized for Osano: org and user scoping, per-owner storage modes, recall timeout tuning, queue boundaries. We did not pretend we were going to out-research the research stack while shipping chat to customers.
Jina embeddings at 1024 dimensions
We wanted multilingual coverage without running separate embedding stacks per locale, and we cared about cost and latency on short memory snippets, which are usually well under 500 tokens.
The operational contract matters more than the benchmark slide: if Jina is down, chat still answers. Recall returns empty. Extraction can still run stateless. Memory is enrichment, not a hard dependency on the critical path. That is a deliberate product decision, not a happy accident.
How it powers chat without adding latency
Walk through a normal question.
- The API calls recall for the user and org scopes.
- Recall races against a timeout (we use about 1.2 seconds). User scope first, then parallel fan-out across a bounded number of org scopes.
- We merge, cap per source, rank by relevance plus recency and stability signals, and truncate to a small character budget for the prompt.
- The agent answers. The user sees response time dominated by the model, not memory.
- After the response, extraction enqueues asynchronously. Facts, episodes, and procedures get routed to layers. Embeddings get written. FSRS initializes where appropriate.
- Consolidation runs on schedules. Replay selects high-value memories. FSRS gets boosts. Edges strengthen or prune. Weak graph connections die quietly offline.
If recall times out, we ship the answer anyway. If the memory subsystem is unhealthy, chat degrades to stateless mode instead of hanging. I would rather miss a helpful snippet than miss an SLA.
That is the line competitors gloss over when they demo "perfect memory" on a single scripted thread.
The algorithms that make it feel cognitive
FSRS. Semantic facts carry difficulty and stability. Reviews schedule from memory dynamics, not a fixed "review every seven days" calendar. Emotional episodes can start with higher stability. Reviews at low retrievability can produce larger stability gains. The open FSRS ecosystem exists because SM-2 style schedulers leave accuracy on the table. We are betting that bet for long-lived user knowledge.
Hebbian graph learning. When two memories co-activate in the same recall, edge weight increases with diminishing returns. Unused edges decay. Very weak edges prune. We normalize so the graph does not turn into a hairball of unit weights. The point is learning which facts travel together in practice, not hand-labeling every relation.
Sleep consolidation. Offline jobs replay a small batch selected by access, emotional weight, recency, and instability. Replay boosts stability and strengthens edges between co-replayed nodes. This is where "important but rarely cited" memories get a fighting chance. It is also the hardest layer to validate in a chat UI, which I will come back to.
Bayesian confidence on epistemic edges. Supports lift confidence. Contradictions cut it. Iteration until convergence gives the agent a way to reason under uncertainty without pretending every extracted fact is equally true.
None of this is deterministic in the way a SQL report is deterministic. That is the point, and also the audit headache.
Multi-tenant customization
We run shared storage with strict owner scoping by default. Every query must include owner identity. Miss a filter and you have a cross-tenant leak. We treat that like a security incident waiting to happen and enforce it in review, tests, and linting.
Per-owner isolated LanceDB directories are rolling out for stronger physical isolation. DSAR deletes become directory wipes instead of row hunts. The cost is adapter lifecycle management, LRU limits on open databases, and migration complexity when you split tenants that used to share a cluster.
Org-scoped recall is deliberately bounded. We will not fan out to hundreds of org databases on every message. Caps keep latency predictable. Merge rules stop one empty scope from starving the context budget.
Honest trade-offs
| Area | What you gain | What it costs |
|---|---|---|
| FSRS scheduling | Recall frequency adapts per fact without manual thresholds | You need recall grades and tuning discipline; debugging "why did it forget?" is harder |
| Seven layers | Preserves cognitive distinctions | More code paths, more incidents, more on-call surface |
| Graph + Hebbian + Bayesian | Patterns strengthen from use | Non-deterministic behavior; harder audits than flat retrieval logs |
| Sleep consolidation | Offline reinforcement for salient memories | Impact is indirect; validation is slow |
| Per-owner storage | Physical isolation reduces SQL scoping risk | Adapter LRU, migration ops, more disks to manage |
| LanceDB | Vectors + structured fields + columnar efficiency | Younger ecosystem; schema migrations are intentional projects |
| Jina + breaker | Multilingual embeddings with graceful degradation | Breaker open means empty recall; you must monitor embedding health |
| Recall timeout | Chat never blocks on memory | High-cardinality recalls may miss slower layers |
The philosophy is simple and unpopular in vendor decks: accept complexity and partial non-determinism if you want memory that behaves plausibly human. A single vector collection is easier to demo, easier to audit, and easier to sell. It also forgets badly.
Versus a "simple vector DB" mental model
| Property | Simple vector DB | Our ZenBrain + LanceDB stack |
|---|---|---|
| Decay | Vectors persist until deleted | Time-based, emotion-modulated, FSRS-scheduled |
| Ranking | Cosine distance | Distance + task context + stability + recency + emotional weight |
| Relations | Flat records | Typed graph edges with learning |
| Learning | Static embeddings | Hebbian co-activation, FSRS updates, Bayesian propagation |
| Consolidation | None | Offline replay and pruning |
| Failure | Query errors or junk hits | Graceful degradation; session buffers still work |
| Multi-tenancy | Row filters | Shared scoping now; per-owner isolation on the roadmap |
| Layer semantics | One bucket | Working, episodic, semantic, procedural, core, cross-context |
If your product only needs "find similar docs," stop reading and use the simple thing. If your product promise includes "this AI knows our compliance context over quarters," the simple thing will betray you quietly.
What is still moving
Per-owner storage is the next isolation milestone. Graph-driven agent decisions that use propagated confidence are future work, not today's default path. Cross-context deduplication will keep getting better as we see more real collisions between work, personal, and learning scopes.
This is a living system. I am telling you how we think, not claiming we finished cognition.
Why we bothered
Most AI products overestimate what models do autonomously and underestimate the engineering to make behavior reliable. Memory is where that gap shows up in user trust. Users forgive a wrong answer once. They do not forgive an assistant that confidently forgets the constraint you set last month.
We chose a harder architecture because privacy and governance customers do not want a chatbot with amnesia dressed up as intelligence. They want continuity with boundaries: what is pinned, what decays, what is org-scoped, what can be deleted cleanly, and what happens when retrieval fails.
I have watched compliance teams try to trust assistants that could not hold a constraint across a quarter. The model was fine. The memory model was a filing cabinet with amnesia. That is why we accepted ZenBrain complexity instead of shipping another "we indexed your PDFs" story.
If you are building memory for agents, ask one question before you pick tools: are you building search, or are you building a lifecycle? Search is solved. Lifecycle is not.
Tell me where you think we overbuilt. I have scars from the simpler path already.