Teach Your Agent to Recall: Searchable Conversation History over MCP

Every chat with an agent starts from zero. On Monday someone reviews the output of two sites with the assistant, writes down who owns which follow-up and settles on a budget case. On Thursday they open a new chat and ask “what did we decide about N-11?”, and the agent has no idea what N-11 is. The user scrolls back, copies, pastes and explains everything again.

This article shows how we closed that gap with very little code. The agent gets one tool, recall_my_conversations, that searches the current user’s own earlier chats and returns dated passages. The search engine is funes, an open-source project we reuse instead of writing our own. The glue is a small FastMCP server, and everything runs in one container.

This is the third piece in a series on agent tooling over MCP. The first gave the agent a safe place to run Python. The second kept bulk data out of the transcript by passing references instead of rows. This one lets the agent look things up in the transcripts themselves.

Recall is not memory, and that is fine

“Agent memory” usually means something ambitious: a system that extracts facts from conversations, keeps a profile up to date, resolves contradictions and quietly feeds the right context into every prompt. That is hard to build and hard to trust. When it goes wrong, the agent “remembers” something you never said, and nobody can see where it came from.

We wanted something much simpler, which works more like a person searching their own email:

• The past stays exactly as it happened. Nothing gets summarised, rewritten or merged.
• The agent decides when to look back, by calling a tool.
• What comes back is verbatim passages with dates, so the agent can quote them and the user can check them.
• If nothing relevant comes back, the agent says so.

Recall gives the agent hints into the past. It doesn’t give it a model of the user. The agent still reasons from scratch every time, but it no longer works from scratch. It can pick up the parameters it used last week, the owner it noted, or the decision that was taken, instead of asking the user to repeat them.

That turns out to be most of the value, at a fraction of the risk.

Don’t reinvent the bicycle: funes

Searching past agent sessions is a solved problem for coding agents. funes, from Hugging Face, is a “durable, searchable memory of your past agent sessions”. It indexes Claude Code, Codex and similar transcripts so the agent can recall earlier work. (We used it while preparing this article to find our own earlier write-ups.)

What makes it a good fit for a product, and not only for a developer laptop:

• One binary, no services. A single Rust executable with an embedded storage engine (Lance). There’s no database server, vector service or queue to run.
• No LLM inside. Retrieval runs on two small local models, the bge-small-en-v1.5 embedder and an optional bge-reranker-base cross-encoder. Nothing leaves the container, and recall can’t hallucinate. It only returns text that exists.
• A memory is a directory. FUNES_HOME=/some/dir gives you an independent memory. One directory per user is all the tenant isolation the storage layer needs.
• A plain input format. It indexes .funes.jsonl files: one line per message with a session id, a sequence number, a timestamp, a role and text blocks. Any chat system can produce that.
• Read verbs made for agents: recall (ranked search), get (read a range of a session), scan (exact string in one session), sessions, sketch.
• Permissive licences: funes is Apache-2.0, both models are MIT.

How recall works inside

A query goes down two paths at once. A dense vector search finds passages with similar meaning (“work order” finds “ticket”). A BM25 full-text search finds exact tokens such as MT-4471 or N-11, which embeddings handle poorly. The two ranked lists are merged with reciprocal rank fusion, and each hit comes back with its neighbouring turns, so the agent sees the question that led to an answer as well as the answer.

We saw the same split in our table-name retrieval experiment. Dense and lexical search fail on different queries, and fusing them is cheap insurance. funes does it out of the box.

The optional third stage is a cross-encoder that re-scores the pool. funes switches it on when the candidate pool is at least four times k. It costs about three seconds and 1.6 GB of RAM per call (numbers below). For conversational recall we keep it off: the pool stays below 4 × k, and the agent compensates by asking again with different words when the first try misses.

The architecture

There are two paths, and they deliberately use different credentials:

• Write path (dashed). The chat gateway already sees every message, so it forwards each finished turn to the recall server over an internal route protected by a shared key. The agent platform isn’t involved. It can’t write to memory, and a prompt injection can’t plant “memories”.
• Read path. The agent calls recall_my_conversations like any other MCP tool. The server takes the user from the caller’s token and searches only that user’s directory. The query text never chooses whose memory is searched.

The agent platform can be anything that speaks MCP. We first built this inside a production copilot running on a commercial agent platform, with a Java MCP server. The demo below is a self-contained Python version of the same design.

Write path: forward finished turns

The gateway posts the whole conversation so far after each turn. That sounds wasteful, but it makes the write idempotent: the server rewrites one file per conversation atomically, and funes, which tracks what it has already embedded, only processes new text.

@mcp.custom_route("/turns", methods=["POST"])
async def ingest(request: Request) -> JSONResponse:
    if not hmac.compare_digest(request.headers.get("x-ingest-key", ""), INGEST_KEY):
        return JSONResponse({"error": "unauthorized"}, status_code=401)
    body = await request.json()
    user, cid = body.get("user"), body.get("conversation_id", "")
    if not user or not re.fullmatch(r"[A-Za-z0-9_-]{1,128}", cid):
        return JSONResponse({"error": "user and a safe conversation_id are required"}, status_code=400)
    payload = to_funes_jsonl(cid, body.get("turns", []))

    src = home(user) / "src"
    src.mkdir(parents=True, exist_ok=True)
    tmp = src / f".{cid}.tmp"
    tmp.write_text(payload)
    tmp.replace(src / f"{cid}.funes.jsonl")   # atomic: recall never sees half a file
    schedule_index(user)
    return JSONResponse({"stored": cid}, status_code=202)

home(user) is DATA / sha256(user)[:24], so directory names don’t reveal user ids. The conversion to funes’ format is a few lines:

def to_funes_jsonl(conversation_id: str, turns: list[dict]) -> str:
    lines = []
    for seq, t in enumerate(turns):
        ts = datetime.fromisoformat(t["ts"].replace("Z", "+00:00")).astimezone(timezone.utc)
        lines.append(json.dumps({
            "session_id": conversation_id, "turn_uuid": t.get("id") or f"{conversation_id}-{seq}",
            "seq": seq, "ts": ts.strftime("%Y-%m-%dT%H:%M:%SZ"), "role": t["role"],
            "harness": "copilot", "blocks": [{"block_type": "text", "text": t["text"]}],
        }))
    return "\n".join(lines) + "\n"

Indexing is debounced per user. A burst of turns produces one funes index run, never two concurrent runs for the same directory, and a write that arrives during a run earns exactly one more:

def schedule_index(user: str) -> None:
    st = index_state.setdefault(user, {"task": None, "dirty": False})
    st["dirty"] = True
    if st["task"] is None or st["task"].done():
        st["task"] = asyncio.create_task(index_loop(user))

async def index_loop(user: str) -> None:
    st = index_state[user]
    while st["dirty"]:
        await asyncio.sleep(INDEX_DEBOUNCE)
        st["dirty"] = False
        await funes(user, "index", "--yes", str(home(user) / "src"), timeout=300)

Recall doesn’t wait for indexing. A turn becomes searchable a few seconds after it closes, which is fine because nobody asks “what did we just say?” through a search tool.

Read path: one tool

@mcp.tool
async def recall_my_conversations(query: str) -> str:
    """Search the current user's own earlier conversations with this assistant.
    Use it when the user refers to something discussed before ("last time", "earlier",
    "what did we decide", "the numbers we used"). Use short queries built around identifiers
    and key nouns ("K-2 machine ticket"); search again with other words if needed.
    Returns dated passages from past chats. Quote them; do not invent details."""
    user = get_access_token().client_id          # from the verified token, never from the query
    if not (home(user) / "memory").exists():
        return "No earlier conversations are stored for this user yet."
    async with recall_slots:
        code, out = await funes(user, "recall", "-k", str(RECALL_K),
                                "--candidates", str(RECALL_CANDIDATES), "--", query[:500],
                                timeout=RECALL_TIMEOUT)
    if code != 0:
        return "Conversation memory is unavailable right now; continue without it."
    out = HINT_LINE.sub("", out).strip()         # drop funes' "→ get <session> --memory <path>" hints
    return out[:MAX_RESULT_CHARS] or "Nothing relevant found in earlier conversations."

A few details matter more than they look:

• The docstring is the trigger and the manual. Agents call a tool when its description matches the user’s phrasing. “Last time”, “earlier” and “what did we decide” are the words people use. The description also says how to query, because short, identifier-heavy queries rank better (see the measurements).
• -- before the query. A user who types -k 100 is searching for that text and isn’t passing a flag.
• Strip the hint lines. funes prints → get <session> --memory <path> after each hit for agents that can call get. Ours can’t, so the lines only leak server paths and session ids into the answer. We saw exactly that in our first trial, where the agent started citing conversation-id prefixes.
• Fail soft. A memory problem must never break the conversation. The tool says memory is unavailable, and the agent carries on.
• Bounded cost. At most four concurrent recalls, a 10-second timeout and a 4,000-character cap on what enters the context.

The funes call itself runs with a cleared environment: FUNES_HOME set to the user’s directory, HF_HUB_OFFLINE=1, and models read from a directory baked into the image.

Packaging: one container

FROM python:3.12-slim

ARG FUNES_VERSION=1.8.0
ARG FUNES_SHA256=3c7eacbaa4487d253f8e13ef7223325ba05069fc258533db6e19bb5765181fea
ADD --checksum=sha256:${FUNES_SHA256} --chmod=0755 \
    https://github.com/huggingface/funes/releases/download/v${FUNES_VERSION}/funes-x86_64-linux /usr/local/bin/funes

# Bake the embedder: a throwaway index + recall downloads exactly what funes needs.
ARG BAKE_RERANKER=0
ENV HF_HOME=/opt/hf
RUN mkdir -p /tmp/warm/src \
 && echo '{"session_id":"warm","turn_uuid":"t0","seq":0,"ts":"2026-01-01T00:00:00Z","role":"user","harness":"copilot","blocks":[{"block_type":"text","text":"warm up"}]}' > /tmp/warm/src/w.funes.jsonl \
 && FUNES_HOME=/tmp/warm/home funes index --yes /tmp/warm/src \
 && FUNES_HOME=/tmp/warm/home funes recall -k 1 --candidates 2 warm \
 && if [ "$BAKE_RERANKER" = 1 ]; then FUNES_HOME=/tmp/warm/home funes recall -k 1 --candidates 8 warm; fi \
 && rm -rf /tmp/warm && chmod -R a+rX /opt/hf

RUN pip install --no-cache-dir fastmcp==4.1.0
WORKDIR /app
COPY recall_server.py .
RUN useradd -u 10001 -m app && mkdir -p /data/memories && chown app /data/memories
USER app
ENV HF_HUB_OFFLINE=1
CMD ["python", "recall_server.py"]

services:
  recall-mcp:
    build: .
    ports: ["8000:8000"]
    environment:
      INGEST_KEY: ${INGEST_KEY:-demo-ingest-key}
      RECALL_K: "3"
      RECALL_CANDIDATES: "8"
    volumes:
      - memories:/data/memories
volumes:
  memories:

The trick worth copying is the warm-up step. Instead of hard-coding model files, the build runs one tiny index and one recall, so funes downloads exactly the models it uses into HF_HOME. At runtime the container never contacts the Hugging Face Hub. The image layers are 218 MB for the funes binary, 134 MB for the embedder and 119 MB for FastMCP. Baking the reranker would add another 1.13 GB, which is one more reason to keep it off. Use ADD --chmod rather than a separate RUN chmod, which copies the 218 MB binary into a second layer.

Does it work? Two weeks in the life of an operations lead

We seeded the memory with six conversations that someone running a few sites might plausibly have over two weeks: an output review of two sites, forecast calculations for a budget, a health check of nightly jobs, a shift handover, some unit-ratio arithmetic and a capacity check. The examples are fictional and deliberately generic; any domain with IDs, owners, tickets and numbers looks the same to recall. A second user, bob, got one conversation about a different site. The seeder plays the chat gateway and posts the conversation after every turn, exactly as production does.

For the agent we used Claude in headless mode, with this server as its only tool and no other knowledge of the domain:

claude -p "$QUESTION" --strict-mcp-config --mcp-config mcp-alice.json \
  --tools "" --allowedTools mcp__recall__recall_my_conversations \
  --system-prompt "$(cat system.md)"

Every answer below was produced by the agent from recalled passages alone.

A decision and its blocker, across two chats. “What did we decide about the N-11 upgrade, and what has to happen before it can be scheduled? Is that blocker still open according to my notes?” The agent made four recalls (N-11 upgrade, N-11 blocker schedule, MT-4471 K-2 machine, K-2 closed) and answered:

Decision (2026-10-08): You decided to go ahead with the N-11 upgrade proposal. You said 91% load is “acceptable but tight.” The expected North finishing line load after the upgrade is about 2,185 of 2,400 units/d (91%). Before it can be scheduled: maintenance ticket MT-4471 (the K-2 machine issue) has to be closed first. Is the blocker still open? My notes don’t show it closed. […] I found nothing after 2026-10-08 that says MT-4471 was resolved or closed. […] Check the ticket in the maintenance system before you schedule.

The decision came from the 10-08 chat and the ticket details from the 10-03 shift handover. The agent joined them, and it was honest about what the notes don’t contain.

A cause found across two chats. “Why were the reconciliation columns empty at the end of September, and who did I escalate it to?” In one chat the user noticed the empty columns and planned to ask Tomas. In a later chat, two failed job runs traced back to data feed F-204. With two recalls the agent connected them: the runs failed on the F-204 feed gap, and the issue went to Priya, the integrations lead. It added that the chat only records the plan to escalate, and that the two failed runs (09-26 and later) don’t explain the empty days from 09-23 to 09-25. It also said that the recalled text cut off the exact error message. That is true: funes returns neighbouring turns as short previews, and the agent said so instead of filling the gap.

A cross-chat to-do list. “Give me one consolidated list of my open follow-ups across all earlier chats.” Five recalls produced seven items from five different chats, each with owner, due date and source date. The agent flagged the overdue ones against today’s date, pointed out that two items were the same underlying problem, and marked a weekday it had turned into a date (“the chat only says Thursday”) so the user could check it.

Exact numbers. “Which forecast case did we pick as base case, with which parameters, and what did it give at 18 months?” Hyperbolic, q0 1,500 units/d, Di 0.25/yr, b 0.6, giving 1,069.5 units/d and 693.3 k units, with the exponential case for comparison. The numbers match the original calculation exactly, and the agent added that the chats record the decision but not the reason for it.

Negative control and isolation. “Did I ever discuss motor P-7 or ticket MT-5120?” As alice: “No. The earlier chats I can search have nothing on motor P-7 or ticket MT-5120.” It listed the closest hits (machine K-2, ticket MT-4471) as a possible mix-up. As bob, the same question returned his South hub chat with the P-7 bearing noise and ticket MT-5120. Alice can’t see bob’s history even when the words match exactly, because her token only ever opens her directory.

Agent runs took 4–11 seconds end to end, with two to five recall calls each. The recalls account for about a second of that. The full answers are in examples/agent-runs.md.

Same result in a production-like stack

Before writing the demo we wired the same design into a real copilot: a commercial agent platform, our own chat gateway, a Java MCP server and funes in the same container. Four working chats (data lookups, calculations, a handover note) were followed by four recall chats in new conversations. The agent made 13 recall calls. Every recalled fact was correct, including a recomputation of a budget case for a longer horizon from recalled parameters, and a negative control about equipment that was never mentioned. One recalled detail came from an answer whose stream our test driver had given up on. The gateway had still forwarded the finished turn, and memory knew more than the test harness did.

Measurements

To get beyond hand-picked examples, we generated a synthetic history for one user: N conversations of six messages, each hiding one checkable fact (a ticket number, a quality-check reject rate, a task owner or a decision reference) among small talk. Distractors are deliberate. Every site and piece of equipment appears in about six conversations (in small talk and in other facts), and the same twelve people, task types and symptoms recur everywhere, so a query has to match a combination and can’t rely on one rare word. Each question has exactly one correct answer. Questions are paraphrased: the fact says “Ticket MT-1040 is open” and the question asks “which work order number was raised for the seal leak on pump P-003?”. A hit means the expected answer appears in the text the tool returns.

Everything ran inside the container, limited to 4 CPUs and 4 GB, with one fresh funes process per call, exactly as the server runs it.

100 questions per size, 4 CPUs. Where a cell shows two values, the benchmark ran twice with a full re-index each time; the spread is run-to-run variance of the index build.

What the numbers say:

• Recall latency doesn’t depend on history size. A recall costs about 0.2 seconds and 240 MB from 1,000 to 10,000 conversations, and most of that is starting the process and loading the embedder. For scale: an engineer who has ten chats a day builds up about 2,500 conversations a year.
• The question is never lost, only ranked below the cut. With eight passages instead of three, every question hit at every size. With three, up to 9% of questions missed. The misses we inspected had one cause in common: near-identical facts about different equipment (“vibration alarm keeps coming back” on K-039, K-019 and K-007) outranked the right one because the long, paraphrased question matched them on meaning, and the distinguishing token was only an ID. The same question shortened to “compressor K-047 ticket” put the right conversation first. That is how agents phrase tool calls anyway (our agent’s queries looked like MT-4471 K-2 compressor closed), and short queries scored a little higher across the board. We added “use short queries built around identifiers” to the tool description.
• The reranker buys back the k=3 misses at a high price. It reached 99–100% at k=3, but it costs 15 times the latency, seven times the memory and a 1.1 GB model in the image. Returning more passages gets the same accuracy at no latency cost and spends context tokens instead. We keep k=3 because agents re-query when the first try doesn’t answer the question. Raise k (and the character cap) if your agent doesn’t.
• Indexing is cheap where it matters. A new conversation becomes searchable within seconds and a few hundred MB, off the request path. Building a 10,000-conversation history from scratch takes 20 minutes and 2 GB. That’s a one-time backfill, so run it as a batch job, not inside the server.
• Disk use is modest but not linear. The memory directory holds vectors, a full-text index and table versions. Its size depends on how often the storage engine has compacted, which is why 3,000 conversations took more space than 10,000 in our runs. Budget hundreds of MB per very heavy user, and a few MB for a typical one.

What this is, and what it is not

Limits

• History isn’t truth. Recall returns what was said, including mistakes and plans that never happened. Our agent handled this well (“your notes don’t say whether it was closed”), but that comes from the agent and its prompt. The tool can’t guarantee it.
• Recall chats are remembered too. The answers from recall conversations are indexed like any other turn. Usually that helps, since a consolidated to-do list becomes searchable. But a wrong recall answer would also become a future “memory”. Excluding turns that were mostly recall output is a reasonable refinement.
• Delete is a rebuild. funes is append-only. “Forget this conversation” means deleting its source file and re-indexing that user’s directory. That’s cheap for one user, but it’s still an explicit operation you need for privacy requests and retention policies.
• Secrets and personal data persist. Anything a user pastes into a chat lands in their memory. Redact at the gateway before forwarding, and decide retention up front. (funes has a scrub command for secrets, not for business data.)
• English-first embeddings. bge-small-en is an English model. BM25 still matches exact codes in any language, but meaning-based search in other languages needs a multilingual embedder.
• Agent behaviour varies. Without the explicit tool name in the request, the agent has to decide to look back. A good tool description gets most of the way there. A line in the system prompt (“if the user refers to earlier work, recall before answering”) gets the rest.
• One process per call. Simple and robust, but every recall pays the process start and model load (part of the ~0.2 s). At very high request rates, a long-running funes MCP server per tenant (funes mcp <memory>) would avoid that.

Wrap-up

Agents don’t need a brain transplant to stop starting from zero. They need a way to look things up in what already happened. With funes doing the retrieval, the whole feature comes down to one route that stores finished turns, one tool that searches them under the caller’s identity, and a container with a pinned binary and a baked model. It isn’t memory in the science-fiction sense. It’s recall: dated, verifiable hints into the past that the agent asks for when it needs them. For the people using the agent, it’s the difference between “remind me what N-11 is” and “you decided to go ahead after MT-4471 closes; it’s still open”.

The complete demo (server, Dockerfile, compose file, seed conversations, agent wrapper and benchmark) is in the companion repository.