My assistant could describe my work but not do any of it, so I replaced the RAG with an agent

I built an interface over my own work history. The first version did what everyone builds: retrieve the twelve most relevant past sessions, stuff them into a prompt, stream the answer back.

It was well engineered and close to useless, for a reason that took me a while to name.

Retrieval can only describe

I reported a bug to it by voice. The dictation mangled the app's name, so what arrived was a run-on sentence about a "dentigo essay app" with no punctuation, complaining that the release button needed a mouse and the text box was too small.

It searched four months of transcripts for the phrase, found nothing, and told me:

No episode in the record covers Dentigo/press-to-release or search-bar sizing. Try searching your history for "Dentigo."

It suggested I search my history for a word that does not exist, invented by a mishearing of its own name.

That is a correct answer to a question nobody asked. I was filing a bug against the app I was typing into. Every input became a history query, because retrieval was the only verb it had.

Swapping the loop, not the model

The Claude Agent SDK runs the same loop as Claude Code: Read, Edit, Bash, Grep, WebSearch, streamed back as they happen. Replacing the endpoint was about eighty lines.

const q = query({
  prompt: text,
  options: {
    cwd: dir,
    permissionMode: "bypassPermissions",
    settingSources: ["user", "project"],
    systemPrompt: { type: "preset", preset: "claude_code", append: APPEND },
    ...(sessionId ? { resume: sessionId } : {}),
  },
});

Three of those lines carry most of the value.

settingSources reads the existing config, which is where my own MCP servers are already registered. Recall over 2,000-odd episodes arrives as a tool the model chooses to call rather than context forced in front of every question. Retrieval became optional, which is what it should always have been.

resume makes follow-ups cheap. Asked a question, then asked which of two files it had read first, it answered from memory of its own previous turn for 1.2 cents and read nothing again.

cwd was the one I got wrong.

Working directory is a starting point, not a boundary

Asked about a change in a different project on the same laptop, it grepped its own repository, found nothing, and reported that it had no access to that codebase.

It was right about its sandbox and wrong about the world. The source was one directory over.

Two things were missing. additionalDirectories never included anything but its own project, and nothing in the prompt named a single other repository. A directory the model does not know about is a directory it will not look in.

Deriving the list needed two sources, because neither is complete on its own. The database knows every project I have actually worked in. A filesystem scan catches repositories I have not opened an agent in yet. A scan alone missed one of the most active projects entirely: its app is a level down inside a subfolder, so the directory a scan sees carries no .git or package.json and looks like any other folder.

Thirteen directories are now in scope and named, and the same question that produced "I have no access" lands in the right repository on the first try, without being told where to look.

There is a second version of the same mistake worth separating out, because I made it too. The agent already had a shell, so it already had every CLI I am logged into on this machine. Nothing about the directory fix changed that. What was missing was a prompt that said so. Both failures are the same shape: a capability it holds and does not know it holds is a capability it will not use.

What it is worth

A three-turn answer that reads two files costs about twelve cents. A follow-up that answers from context costs one. That is more than a retrieval call and less than my time.

A retrieval endpoint and an agent are different products behind the same interface. One answers questions about what happened; the other changes what is happening. I spent a week building the first before noticing I wanted the second.

Start with the tools and add retrieval as one of them. Going the other way costs you a rewrite.