# The memory palace: how fresh AI agents start with my rules

**Topic:** agent memory · **Published:** October 2026 · **Reading time:** 10 min

Every AI agent session starts blank. So does every session after context compaction, when the tool shrinks a long chat into a short summary. The agent has the model's skills and nothing else. It does not know my rules, the state of the project or what it was doing ten minutes ago.

I run many agents on a regulated crypto platform, so this happens all day. I built a memory system for it and call it the memory palace. It is plain markdown files, a few scripts and a handful of hooks. It has no database and no service.

## The problem

Without memory, a fresh agent does one of three things:

- **It re-derives facts.** It says a fact is missing when it was written down two days earlier. One agent told me that five times.
- **It repeats old mistakes.** A rule I gave once is gone after the next restart.
- **It trusts stale chat memory.** It "remembers" a state that changed since, and acts on it.

Before I built the palace in early August 2026, I had tried an automatic memory tool and removed it. In two days it had stored 14 prompts and no observations, and it still reported itself healthy. So the palace follows one design rule: plain files that anyone can read with `ls` and `grep`.

## The layers

I do not load everything into every session. The palace has layers, and each loads at a different time.

1. **A binding rules file.** One file at the workspace root loads in every session. It holds the rules that always apply. It has a size cap: it stays under 32 KiB, because the Codex agent truncates a longer file without warning. Today it is 27,491 bytes. When a section grows, I move the full text out and leave one binding sentence.
2. **A references index.** A short table says "read this file when this happens". The triggers are things like delegating work, landing a change or "after a verified outcome that should outlive the session". Each reference file also carries two lines that say when to read it and why it exists. An agent loads a reference only when its trigger matches.
3. **Project records.** Each project has a README, a state file, a task list, a decisions file and an audit log. They say what the project is and what happened to it.
4. **The palace itself.** A folder of notes, one fact per note.
5. **Task checkpoints.** One small file per lane or task, with its next step.
6. **An inbox.** Drafts that were captured automatically and are not yet rules.

The rules file says how to behave. The project records say what exists. The palace says what I learned. The checkpoint says where this task stands.

## How a note is written

A note holds one fact. It starts with a short header: a name, a one-line description and a type. The three types are feedback (a rule about how to work), project (a fact about a system) and reference (where to find something). A feedback note's body has three parts:

- **The rule** in one or two sentences.
- **Why**, with my words and the date.
- **How to apply**, so the next agent knows when the note matters.

I keep my own words in the note. For example, one note says that a finding is not done until a mechanism stops it from happening again. Its "why" part quotes me on 12 August 2026: "create a system for these do not just note it and then leave it". An agent that reads the quote understands the intent and applies the rule to cases the note never mentioned.

The date lets a later agent judge whether a note is still current.

## The index is generated, never edited

The palace holds about 595 notes. Nobody can read 595 notes at startup, so one index lists them. The index is one line per note: a link and the description. A script builds it from the notes themselves. The first lines of the index say "do not hand-edit below".

Today the index has 614 lines and 110,862 bytes. It groups notes into rooms by type, plus a final section for superseded notes. Agents search it for the topic at hand, then open the matching notes.

The script has three commands:

- **index** rebuilds the index, the backlinks and the rooms.
- **ask** searches the palace and the standing records and returns one answer, with sources in order of authority.
- **check** looks for orphan notes, broken links and missing headers, and fails if the palace is rotten.

The script's comments record an early bug. The first index generator matched nothing, because the header keys were indented, and it rebuilt the index empty over a good one. A generator that can silently produce nothing is the same defect as a test that cannot fail.

## The startup routine

At startup, after compaction and when the task changes owner, the agent runs the same routine:

1. Read the workspace state file.
2. Find the project in the registry.
3. Read the project README and any local rules.
4. Read the project's state, decisions, tasks and the latest audit-log entries.
5. Verify any fact that can change against the real repository or service.

Step 5 matters most. A note says what was true when someone wrote it, so the agent checks the real system before acting on anything that can change.

## The hook that does not trust the agent to remember

A rule that depends on the agent remembering to read the palace fails on the day the agent forgets. So a hook runs on every prompt. It injects three things:

- The path of the palace index, with the instruction to read matching notes before re-deriving a fact.
- The path of the agent's own checkpoint file, with its size and last update time.
- One rule: record each request from me, word for word, in the checkpoint before starting work on it.

After compaction, my original words are still on disk.

A second hook runs when the agent finishes a turn. If the turn changed files but not the checkpoint, it warns me and the agent. It never blocks anything. Every path in both hooks exits cleanly, so a bug in the hook cannot stop a prompt.

The comments in an older session-start hook explain why. On 6 August 2026 I wrote: "IF YOU ARE UNSURE YOU LOOK FOR PALACE." The palace was referenced but never present in the context. A pointer that someone must follow by hand gets skipped, and the agents drift.

## Checkpoints for work in flight

Notes hold lasting knowledge. A checkpoint holds the state of one task. It records the objective, the environment, where the work stands, the exact next action, the checks that remain, the sources and the risks. An agent saves one after a significant change, when it waits or hands over, and before it ends or compacts.

The store keeps numbered revisions and never overwrites an old one. Only one session owns a task at a time, and a transfer needs an explicit release and claim. At the moment the task store holds about 30 checkpoints.

Curated notes and the project task list stay authoritative. An observation recorded by a hook is marked unverified.

## The inbox

Automatic capture is useful and risky. A script captures what I say in chat as a draft note. Each draft says it is unreviewed and "NOT a palace rule" until a reviewer rewrites it and moves it into the palace. The inbox holds 11 drafts now. The agents may read them, but may not cite them as rules.

## One place for everything

Memory had spread across more than one store. On 24 September 2026 I told the agents to "unify the memory systems so all is in the same place". The notes from the other store moved into the palace, marked with the date and their source. The old store is now a read-only rollback copy that agents must not read or write.

## What went wrong, and the rules that followed

- **Stale caches.** A cached copy can look current and be wrong. The rule: a stale cache grants no authority. An agent never claims a task, transfers it or marks it done from a stale cache.
- **Recorded, but not enforced.** I had told agents the same rule before, and a note existed, but nothing acted on it. The rule: for every finding, build the mechanism that stops a repeat. The per-prompt hook below is that kind of mechanism: it acts on every prompt, so nobody has to remember.
- **Memory instead of the system.** An agent answered from what it remembered. The rule: verify mutable facts against the repository or service.

## Limits and trade-offs

- **The index is big.** At about 110 KB, it is too large to load whole. Agents depend on search to find the right note.
- **Notes go stale.** Someone must still retire old notes. The index has a section for superseded ones.
- **The hooks warn, they do not block.** I chose that on purpose, because a hook that blocks work causes its own failures. The cost is that an agent can ignore a warning.
- **It needs discipline.** A note exists only if an agent writes it after a verified outcome.

## What I would tell a team starting with agents

1. **Write one fact per note, with the reason and the date.**
2. **Keep the always-loaded file small, with a hard cap.** Move detail behind a trigger.
3. **Generate the index and check it.** Never edit it by hand.
4. **Put the reminder in a hook, not in the agent's good intentions.**
5. **Save the next step before every pause.**
6. **Verify anything that can change.** Memory tells the agent where to look, not what is true today.

Want a memory setup like this in your repository? Email [hello@jeffkent.dev](mailto:hello@jeffkent.dev).