Hindsight: Open-Source Agent Memory with Retain, Recall, Reflect

Most agent-memory tools answer one question: what did the user say before? Hindsight, from Vectorize, asks a second one: what should the agent believe now because of it? Its README describes a memory system built so agents “learn, not just remember”, organized around three operations: retain, recall and reflect.

This is a comparison-minded guide, not a memory 101. I walk through the pipeline in 4 steps, the Docker and pip quickstart copied from the README, a fact-checked table against Mem0 and OpenViking, and when you should skip it.

1. What Hindsight Is: Stars, License, and the Idea

Hindsight is an agent memory system built by Vectorize.io and published as vectorize-io/hindsight on GitHub. The repository was created on 2025-10-30 and, on 2026-10-14, shows 49.7k stars (49,680 in the repo metadata), about 6.1k forks and an MIT license. The latest release is v1.0.0, published on 2026-10-13, after the four 0.10.x releases listed between 14 September and 8 October.

Repo Facts (verified 2026-10-14)

Repository: github.com/vectorize-io/hindsight
Stars: 49.7k on the repo page · repository created 2025-10-30
License: MIT
Latest release: v1.0.0 (2026-10-13)
Server install: pip install hindsight-api (PyPI, Python 3.11+) · Docker image ghcr.io/vectorize-io/hindsight
Backing: Vectorize.io · self-hosted or managed Hindsight Cloud

The README’s pitch is that most memory systems focus on recalling conversation history, while Hindsight is “focused on making agents that learn”. Concretely it keeps four kinds of memory in isolated banks: world facts, experiences, observations (consolidated, evidence-backed beliefs) and mental models. You write with retain, fetch with recall and reason over what is stored with reflect.

Read the claims as claims

Sentences like “the most accurate agent memory system ever tested” and “eliminates the shortcomings of RAG and knowledge graph” are vendor claims. The README states that Hindsight’s numbers were independently reproduced by researchers at Virginia Tech’s Sanghani Center and The Washington Post, and that other vendors’ scores in its chart are self-reported. I could not verify that reproduction myself, so every number below carries its source.

2. Architecture in 4 Steps: Retain, Consolidate, Recall, Reflect

This is how the README and the developer docs describe the flow. Parameter names and defaults come from the docs pages on retain, recall and reflect, read on 2026-10-14.

📥

Step 1 — retain() extracts facts with an LLM

Content goes in as a string or an ordered list of blocks. An LLM extracts facts, entities, relationships and time data, classifies each fact as experience (the bank’s own agent) or world, and normalizes everything into canonical entities, time series and search indexes. retain_extraction_mode defaults to concise; retain_mission steers what gets kept.

🧬

Step 2 — background consolidation into observations

After retain returns, related facts are consolidated into observations: deduplicated beliefs that keep their supporting evidence (exact quotes and a proof count) and are refined, not overwritten, when new evidence arrives. Mental models are standing answers to questions you define; the README says reading one is a database read with no retrieval and no LLM call.

🔍

Step 3 — recall() runs four strategies in parallel

Semantic, keyword (BM25-style), graph traversal and temporal filtering run side by side. Results are merged with reciprocal rank fusion (k = 60 per the docs), reranked by a cross-encoder and packed into a token budget (max_tokens, default 4096). The budget parameter (low, mid, high) sets depth: 100, 300 (default) or 1000 candidates per strategy.

🧠

Step 4 — reflect() reasons in a tool-calling loop

Per the docs, reflect checks mental models first, then observations, and falls back to raw recall. It runs up to 10 iterations and may only cite memory ids it actually retrieved. Bank-level disposition traits (skepticism, literalism, empathy, each 1 to 5) tilt interpretation, and an optional response_schema returns structured JSON.

Benchmarks: sourced vs self-reported

The paper (arXiv 2512.12818, 14 Dec 2025) reports an open-source 20B backbone going from 39% to 83.6% accuracy against a full-context baseline with the same backbone, and larger backbones reaching 91.4% on LongMemEval and 89.61% on LoCoMo. The project’s benchmarks site, fetched 2026-10-14, lists 94.6% on LongMemEvalS and 92% on LoCoMo10, but that page states no models, latency, cost or methodology. Mem0 and OpenViking publish their own numbers on different harnesses, so do not rank them in one leaderboard.

3. Quickstart: A Memory Bank in 5 Minutes (README-Verified)

Everything below is copied from the README’s Quick Start; I verified it against the README on 2026-10-14. I did not execute these commands for this post, so confirm the first run yourself. Pick one server path, then connect a client.

Option 1 — Docker (the README’s recommended path): API on :8888, UI on :9999

export OPENAI_API_KEY=sk-xxx

docker run -it --pull always --name hindsight --restart unless-stopped -p 8888:8888 -p 9999:9999 \
  -e HINDSIGHT_API_LLM_API_KEY=$OPENAI_API_KEY \
  -v hindsight-data:/home/hindsight/.pg0 \
  ghcr.io/vectorize-io/hindsight:latest

Option 2 — bare metal with pip

pip install hindsight-api
export HINDSIGHT_API_LLM_API_KEY=sk-xxx

hindsight-api

Connect a client (Python) and call the three operations

pip install hindsight-client -U

from hindsight_client import Hindsight

client = Hindsight(base_url="http://localhost:8888")

# Retain: Store information
client.retain(bank_id="my-bank", content="Alice works at Google as a software engineer")

# Recall: Search memories
client.recall(bank_id="my-bank", query="What does Alice do?")

# Reflect: Generate disposition-aware response
client.reflect(bank_id="my-bank", query="Tell me about Alice")

Hindsight works with 25+ LLM providers through HINDSIGHT_API_LLM_PROVIDER, including local ones (ollama, lmstudio, llamacpp). The README lists three more entry points: a LiteLLM-based wrapper (pip install hindsight-litellm, then wrap_openai() or wrap_anthropic()), a built-in MCP endpoint per bank at http://localhost:8888/mcp/{bank_id}/, and an embedded Python mode (pip install hindsight-all -U) that starts a server in-process.

Tip: one bank per user or project, tags for scoping

Banks are strictly isolated, with no cross-bank leakage per the README, so one bank per user, agent or project is the safe default. The FAQ adds two traps: per-user banks cannot do cross-user analysis (use one bank with tags if you need it), and metadata is not a filter, tags are. Decide the tag scheme before the first retain().

Your content goes to whichever LLM you configure

Retain sends what you store to the configured model provider, and the README documents Memory Defense (secrets and PII redaction or blocking against 45 patterns) as opt-in per bank. Turn it on and test it, or use a local provider, before you retain anything sensitive.

4. Use Cases: Where Retain, Recall and Reflect Earn Their Keep

The README is explicit about the sweet spot: agents that need a blend of features, such as AI employees that handle open-ended tasks and change behavior based on feedback. These are the four patterns it and the docs lean on.

🧑‍💼

Per-user assistants

The README’s simplest case: retain user inputs and tool calls with tags or metadata, then filter recall per user. Banks give you isolation; observations keep preferences from becoming a flat pile of duplicates.

💻

Project memory for coding agents

A separate package builds a per-repo bank from git history and past sessions and injects it when the agent starts. It lists Claude Code, Codex CLI, Cursor CLI, opencode and others. Remember that your repo history then flows through the configured LLM.

🧭

Agents that learn from feedback

The README’s reflect examples: a project manager reflecting on risks, a sales agent on why some outreach got replies, a support agent on documentation gaps. Mental models keep those answers fresh in the background.

🔌

Drop-in memory for an existing stack

The LiteLLM wrapper adds memory by swapping the client, the MCP endpoint exposes the three tools to any MCP client, and the README counts 60+ integrations across coding agents, frameworks and no-code tools.

One line from the README is worth keeping: Hindsight can be used with simple workflows built with n8n and similar tools, but “may be overkill” for them. If your agent talks to a user once and forgets, you do not need any of this.

5. Hindsight vs Mem0 vs OpenViking: What I Can Verify

Same category, different bets. The table is limited to facts I could read in each project’s README or docs on 2026-10-14. Star counts are rounded from the repository metadata of that day.

Aspect	Hindsight	Mem0	OpenViking
Repo · stars (2026-10-14)	vectorize-io/hindsight · ~49.7k	mem0ai/mem0 · ~67.2k	volcengine/OpenViking · ~39.9k
Repo created	2025-10-30	2023-06-20	2026-01-05
License	MIT	Apache 2.0	AGPLv3 main project · Apache 2.0 for crates/ov_cli and examples
Core model	Banks with four memory types · retain / recall / reflect	add() / search() with user, session and agent state	Virtual filesystem under viking:// for resources, memories and skills · L0/L1/L2 tiers
Retrieval (per docs)	Semantic + keyword + graph + temporal in parallel, RRF fusion, cross-encoder rerank	Single pass: semantic, BM25 keyword and entity matching fused (April 2026 algorithm)	Directory-scoped find / search; summaries (L0, L1) before full content (L2)
Reasoning step	reflect(): tool-calling loop with disposition traits	README documents add and search; no separate reasoning operation	Session commit extracts Markdown memories; ov compile builds wiki, graph or report (with VikingBot)
Needs an LLM	Yes · 25+ providers incl. local (ollama, lmstudio, llamacpp)	Yes · gpt-5-mini is the README default	Yes · a model provider with an embedding model and a VLM
Self-host path	docker run · pip install hindsight-api · Helm · compose with external Postgres	pip library · server via docker compose (cd server && make bootstrap)	uv tool install openviking --upgrade && openviking-server init
Managed option	Hindsight Cloud (usage-based)	Mem0 Platform (app.mem0.ai)	OpenViking Service hosted by Volcengine
Headline benchmark (self-reported)	LongMemEvalS 94.6%, LoCoMo10 92% (project site) · paper: 91.4% / 89.61%	LoCoMo 92.5, LongMemEval 94.4 (April 2026, managed platform with proprietary optimizations)	LoCoMo 80 to 83% accuracy across three agent integrations (README, v0.3.22, Doubao models)

Benchmarks use different harnesses, backbone models and, in Mem0’s case, a managed stack, so the last row is not comparable across columns. I did not run the three systems on one workload.

How I would choose

Hindsight when the agent has to form and revise beliefs, with time-aware or multi-hop questions and a reasoning step. Mem0 when you want the smallest API surface and the largest ecosystem. OpenViking when you want memory, knowledge and skills as an inspectable filesystem and can live with AGPLv3 on the core. It is a judgment from the docs, not a bake-off.

6. When NOT to Use Hindsight: Honest Limits

Memory that reasons costs more than memory that stores. These are the limits I would check before committing, each tied to the project’s own docs.

❌ Reach for something else when

• You do not want an LLM on every write: retain extracts with a model, and the models page says models outside its tested list should support at least 65,000 output tokens for reliable extraction (smaller limits mean lowering HINDSIGHT_API_RETAIN_MAX_COMPLETION_TOKENS). Estimate tokens before bulk-ingesting history.
• You need deterministic writes: the retain docs say extraction is not fully deterministic, a mission that excludes everything yields a document with zero memories, and very dense pages are sampled rather than fully extracted.
• You have a tight latency budget: the FAQ cites about 50 to 100 ms for recall without reranking, 200 to 500 ms with it and 1 to 10 s for reflect. Those are the project’s figures, not mine.
• Your task is cited answers over a static corpus: the project’s own RAG-vs-memory page recommends plain RAG for document Q&A and search with no temporal needs.
• You upgrade without reading the release notes: v1.0.0 lists breaking changes (tag-scoped callers are refused on bank-wide operations, load shedding returns 503 with Retry-After instead of 500, the OpenClaw agent-sdk package is gone) and the release page has no migration guide.
• You plan to wire it into a coding agent unreviewed: the coding-agents package wires itself into your agents and ingests git history and past sessions. Read what it changes before running any npx that touches your agent.

✅ Hindsight is a great fit when

• The agent must revise beliefs as evidence changes (observations with proof counts, refined not overwritten).
• Questions have a time or multi-hop shape: “what happened in June?”, “who else worked with her?”.
• You want one self-hostable service with an MCP endpoint, Prometheus metrics and your own Postgres.
• You already run several model providers and want to choose per operation, including local models.

Golden rule

Treat every benchmark in this space, Hindsight’s included, as a hypothesis. Take 50 real conversations from your product, retain them, and compare recall and reflect answers against your current memory layer and a plain RAG baseline. If you cannot define that eval, you are choosing on README claims.

Conclusion

Hindsight’s bet is that agent memory is a reasoning problem, not just a retrieval problem, and the retain, recall, reflect split makes that bet concrete and easy to try: one Docker command, three method calls, an MCP endpoint. The first 1.0 landed on 2026-10-13, so expect the API to keep moving.

Run it on a throwaway bank, measure token cost on your own data, turn on Memory Defense if anything sensitive passes through, and decide with your own eval rather than anyone’s chart.

Hindsight: Agent Memory That Retains, Recalls and Reflects