My agents kept forgetting. So I built them a memory stack.
Four layers, each solving a different failure mode: a lossless context engine that never truly loses a message, structured long-term recall, cross-session memory for a second agent, and a git-backed knowledge vault as the source of truth. The infrastructure that turns a stateless model into an agent that remembers.
A model with no memory re-learns everything, every session.
OpenClaw’s built-in memory is plain Markdown — a MEMORY.md file plus daily logs — and it loses context on long threads.
“We’re losing context every interaction; it’s not getting stored or committed to memory.” — the problem, in my own words at the time.
Every long conversation blew past the context window and the useful detail vanished. An agent with no reliable memory contradicts itself, forgets decisions, and can’t be trusted to run unattended work.
I wanted agents that retain everything long-term — so I set out to layer real memory infrastructure on top of the runtime instead of accepting the default amnesia.
Memory as a stack of layers, not one magic tool.
I had Grok pull the current state of the OpenClaw memory ecosystem, read the underlying source docs directly, and picked components by tier — each layer with a distinct job.
The design principle I locked into agent memory was an integration doctrine: native plugin first, then MCP, then raw API — one capability, one primary integration, lightweight fallbacks, upgrade only when justified.
The rule that keeps it honest: the vault and knowledge layer do not replace runtime memory. Search finds things, the context engine keeps the transcript, structured memory remembers conversations, and the vault holds compiled understanding that compounds over time.
Search, transcript, structured memory, and compiled knowledge are four different jobs. I built a layer for each — not one tool pretending to do all of them.
— Design principle · layer, not a toolFour layers, three machines.
Fast/short-term at the top, durable/long-term at the bottom — each layer an open-source tool integrated into the runtime, mapped across three networked Mac minis and two agent runtimes. GBrain is held as a possible future derived index, not core.
Four layers, fast to durable: a lossless context engine keeps the raw transcript recoverable, structured long-term memory auto-captures and auto-recalls in embedded-local mode, cross-session memory persists the Hermes agent across restarts, and a git-backed Markdown vault holds compiled understanding as the source of truth. It runs across three Mac minis on a Tailscale mesh with a Thunderbolt bridge. GBrain is deferred — held as a possible future derived index, not adopted as core.
Which tool owns which layer.
Each layer is an open-source tool I selected, integrated, and operated. The build is the architecture and integration — not authorship of these tools.
| Layer | Tool | Role | Mode |
|---|---|---|---|
| Lossless context | Lossless Claw martian-engineering | DAG-based lossless summarization; keeps raw messages in SQLite (lcm_grep / lcm_describe / lcm_expand) | OpenClaw extension plugin |
| Structured long-term | Hindsight hindsight.vectorize.io | Auto-capture / auto-recall / reflect on salient facts | Embedded local Postgres + embeddings |
| Cross-session | Honcho docs.honcho.dev | Persist the Hermes agent across restarts | Hybrid mode |
| Durable knowledge | Obsidian + git vault holisticseo/agentic-brain | Markdown source of truth; compiled understanding compounds | Git-backed · GitHub sync |
| Company brain (deferred) | GBrain garrytan/gbrain | Heavier graph + hybrid retrieval over a Markdown repo | Candidate future index — health responding, not core |
Attribution: Lossless Claw, Hindsight, Honcho, and GBrain are third-party open-source projects. This work is the layered architecture, the integration doctrine, and the multi-machine operation — not authorship of the tools. The LLM Wiki pattern (durable layer) is Andrej Karpathy’s; GBrain is Garry Tan’s.
The shape of an internal build.
Internal infrastructure — the "numbers" are components and machines, not client outcomes.
The doctrine that governs the whole thing was locked into MEMORY.md on 2026-06-03: native plugin > MCP > raw API. One capability, one primary integration, lightweight fallbacks, upgrade only when justified — the rule the agents now follow when a new memory capability is added.
Built and running — and honestly, still iterating.
Verified running via live health checks, not claimed. This is recent internal infrastructure, framed as such.
A standalone, durable memory service.
- Lossless-claw and Hindsight installed as OpenClaw plugins and reachable.
- Hindsight healthy with its database connected.
- Honcho connected in hybrid mode for the Hermes agent.
- GBrain health responding on the host machine.
- During a May 2026 decommission of an agent’s communication routes, the memory infra was deliberately kept alive and re-verified — proving it’s a standalone service, not a chat feature.
Recent, in-progress, supervised.
- This is internal infrastructure, not a finished product or client deliverable.
- GBrain is intentionally deferred — a possible future derived index, not adopted as core.
- Hardening items remained open — e.g. an unpinned lossless-claw version flagged as supply-chain hygiene.
Framed exactly right: "built and running," not "productized." The value is the architecture and the operating discipline on a live system.
Default runtime vs the layered stack.
I was the systems architect.
I identified the problem, sourced the component landscape, and made the architecture calls — which tool owns which layer, embedded-local over cloud, GBrain deferred rather than force-fit.
I wrote the integration doctrine the agents now follow, directed the installs machine-by-machine (including unblocking Homebrew permissions on the Hermes mini so Honcho could install), and set the guardrail that the vault — not any single tool — is the durable source of truth. The agents did the install work under my direction; the design and sequencing were mine.