Agent Memory POC PRD — Session Log Ingestion & Summarization

PRD.md

PRD: Agent Memory POC — Session Log Ingestion & Summarization

Status: Draft
Author: Josh Mock
Date: 2026-05-22
Related RFC: RFC.md

---

Overview

This POC validates a small slice of the RFC's "context engine" vision: surfacing useful, durable knowledge from past agent sessions. Rather than building a full graph-based crawler, this POC focuses on the memory tier — ingesting raw session logs into Elasticsearch and using a local LLM to distill them into searchable, agent-readable summaries.

Two standalone TypeScript scripts implement the full pipeline. No framework, no daemon, no watcher — just two scripts you run manually to prove the idea works.

---

Goals

• Confirm that pi session logs contain actionable signal worth remembering across sessions.
• Establish a simple memory-raw → memory-summary pipeline as a concrete foundation for the broader RFC.
• Produce a document schema that can later be extended with embeddings and semantic search.

Non-goals

• Real-time or continuous ingestion.
• Serving summaries back into an active agent session (retrieval is out of scope for this POC).
• Multi-user or multi-machine support.
• Building embeddings or vector search (addressed in recommendations below).

---

Infrastructure Assumptions

• An Elasticsearch instance is already running and accessible.
• Connection is configured via two environment variables:
  • ES_URL — full URL to the Elasticsearch instance (e.g. https://my-deployment.es.us-east-1.aws.elastic.cloud)
  • ES_API_KEY — an API key with read/write access to both target indices
• A local llama.cpp server is running and accessible at http://localhost:8080 (or configurable via LLAMA_URL), hosting a model capable of instruction following (e.g. Llama 3, Mistral, etc.).
• Session logs are stored at ~/.pi/agent/sessions/ in the format pi writes them.
• Both scripts are run manually from the repo root via npx ts-node or equivalent.

---

Script 1: ingest-sessions.ts

Purpose

Reads every session log file in ~/.pi/agent/sessions/, transforms each message into a flat document, and bulk-indexes them into the memory-raw Elasticsearch index. Idempotent: re-running will upsert existing documents without duplication.

Input

Session log files at ~/.pi/agent/sessions/. The script should handle whatever format pi writes (inspect a few live files before implementing to confirm structure).

Output: memory-raw index

One document per message (not per turn). Messages in the same conversational exchange share a turn_id.

Document schema

Field	Type	Description
doc_id	keyword	Stable unique ID: {session_id}_{turn_id}_{message_index}
session_id	keyword	Identifier for the session file (e.g. filename without extension)
turn_id	keyword	Identifier grouping all messages in a single turn (e.g. {session_id}_{turn_index})
turn_index	integer	Zero-based position of this turn within the session
message_index	integer	Zero-based position of this message within the turn
role	keyword	user, assistant, tool, system, etc.
content	text	Full message text content
tool_name	keyword	If the message is a tool call/result, the tool name; otherwise null
session_started_at	date	Timestamp of the first message in the session
message_timestamp	date	Timestamp of this message, if available in the log
model	keyword	Model identifier used in this session, if present in the log

Index as keyword + text (multi-field) for any field you might want both to filter and to search.

Behavior

• Walk ~/.pi/agent/sessions/ and collect all session files.
• Parse each file. Assign session_id from the filename.
• Assign turn_id by grouping messages into turns. The grouping strategy depends on the session log format — a "turn" is typically one user message plus all the assistant/tool messages that respond to it.
• Use Elasticsearch's bulk API (@elastic/elasticsearch client) to upsert documents, using doc_id as the _id.
• Log progress to stdout: files processed, documents indexed, errors.
• Do not crash the entire run on a single malformed file — log the error and continue.

---

Script 2: summarize-memories.ts

Purpose

Iterates through all documents in memory-raw, uses a local LLM to identify ideas worth preserving as long-term agent memory, and writes summarized memory documents to the memory-summary index.

Approach

• Scroll through memory-raw grouped by turn_id (reconstruct full turns, not isolated messages).
• Send each turn's content to the LLM with a prompt asking it to identify any ideas an agent would want to remember for future sessions.
• If the LLM identifies no memorable ideas in a turn, skip it.
• If it identifies one or more ideas, write one memory-summary document per idea.

LLM configuration

• Inference endpoint: local llama.cpp server at http://localhost:8080/v1 (OpenAI-compatible API), configurable via LLAMA_URL.
• Use the /v1/chat/completions endpoint with a structured output prompt.
• Model ID is not hardcoded — pass whatever model name llama.cpp reports or configure via LLAMA_MODEL.

Prompt design

The prompt should instruct the model to return a JSON array. Each element represents one memorable idea found in the turn, with:

• summary: 1-3 sentence description of the idea
• tags: array of keywords (3-8), lowercase, hyphen-separated where multi-word (e.g. typescript, best-practice, repo-elasticsearch-js)

If there are no memorable ideas, return an empty array [].

The prompt must include the full reconstructed turn text (all messages concatenated with role labels), and request strict JSON output.

Output: memory-summary index

Document schema

Field	Type	Description
doc_id	keyword	Stable unique ID: hash of {session_id}_{turn_id}_{summary_text}
summary	text	The 1-3 sentence memory summary
tags	keyword	Array of keyword tags assigned by the LLM
source_refs	object[]	Array of { session_id, turn_id } pairs pointing back to the raw turn(s) in memory-raw
established_at	date	Timestamp of the original turn's messages (use message_timestamp from the raw doc, falling back to session_started_at)
summarized_at	date	Wall-clock time when this summary document was generated
model	keyword	LLM model identifier used to generate the summary

Behavior

• Use the ES scroll API (or point-in-time + search_after) to page through all documents in memory-raw.
• Group documents by turn_id before sending to LLM.
• Use Elasticsearch bulk upsert with doc_id as _id so re-runs are idempotent.
• Concurrency: process turns sequentially to avoid overwhelming the local LLM server.
• Log progress: turns processed, memories written, turns skipped.
• Do not crash on LLM errors — log and continue.

---

File / Project Structure

dev-context-engine/
  src/
    ingest-sessions.ts
    summarize-memories.ts
  package.json
  tsconfig.json
  .env.example

Dependencies

• @elastic/elasticsearch — official ES client
• typescript, ts-node — runtime
• No additional dependencies; use Node's built-in fetch for LLM calls (Node 18+).

.env.example

ES_URL=https://your-deployment.es.us-east-1.aws.elastic.cloud
ES_API_KEY=your_api_key_here
LLAMA_URL=http://localhost:8080
LLAMA_MODEL=llama3

---

Acceptance Criteria

• ingest-sessions.ts runs to completion against a real ~/.pi/agent/sessions/ directory and documents appear in memory-raw.
• Each document in memory-raw has the correct turn_id grouping — you can query GET memory-raw/_search filtered by turn_id and see all messages from that turn.
• summarize-memories.ts runs to completion and writes at least a handful of summary documents to memory-summary.
• Summary documents include valid source_refs that link back to existing session_id + turn_id pairs in memory-raw.
• Re-running either script does not create duplicate documents.
• Both scripts handle an empty sessions directory or empty memory-raw index gracefully (exit cleanly with a log message).

---

Recommendations: Elasticsearch-Native Enhancements

Once the POC works, these Elasticsearch features would meaningfully improve the pipeline:

1. Jina AI inference endpoint (Elastic Inference Service)

Elastic hosts jina-embeddings-v5-text-small-clustering directly via the Elastic Inference Service (EIS). Rather than running the model locally via llama.cpp, you can register it as an inference endpoint in Elasticsearch and have ES generate embeddings automatically via an ingest pipeline. This eliminates the local llama.cpp dependency for embedding generation (though you would still need a local LLM for the summarization step).

PUT _inference/text_embedding/jina-clustering
{
  "service": "jinaai",
  "service_settings": {
    "model_id": "jina-embeddings-v5-text-small-clustering",
    "api_key": "<your-jina-key>"
  }
}

2. Semantic text field + ingest pipeline auto-embedding

On Elasticsearch 8.11+ and Serverless, the semantic_text field type auto-generates embeddings at index time using a registered inference endpoint. Add a summary_embedding field of type semantic_text to the memory-summary mapping, point it at the Jina inference endpoint, and Elasticsearch will embed every summary automatically with no client-side code.

PUT memory-summary
{
  "mappings": {
    "properties": {
      "summary": { "type": "text" },
      "summary_embedding": {
        "type": "semantic_text",
        "inference_id": "jina-clustering"
      }
    }
  }
}

3. Vector similarity search for retrieval

Once embeddings exist, you can retrieve the most semantically relevant memories for a given agent task using a knn query against the summary_embedding field. This is the retrieval step the RFC envisions — before a session starts, embed the user's first prompt and find the top-N matching memories to inject into context.

4. Elasticsearch Serverless

Serverless eliminates cluster sizing and index lifecycle management. For this use case (infrequent writes, occasional reads), the consumption-based billing model is more cost-efficient than a provisioned cluster. The inference API and semantic text features are fully available on Serverless.

5. Inference enrichment processor

For the raw ingest pipeline, an inference enrich processor can classify or tag messages at index time using a trained model — potentially auto-detecting "memorable" content without a separate summarization pass.

---

Technical Notes: Using Jina Embeddings (for newcomers)

What is an embedding?

An embedding is a fixed-length array of floating point numbers (a "vector") that represents the semantic meaning of a piece of text. Two texts that mean similar things will have vectors that are close together in space, regardless of the exact words used. This is what powers semantic search: instead of matching keywords, you match meaning.

This model's characteristics

jina-embeddings-v5-text-small-clustering produces vectors of 1024 dimensions (or smaller via Matryoshka truncation: 32, 64, 128, 256, 512, 768, or 1024). It uses last-token pooling and supports sequences up to 32,768 tokens. It is optimized for clustering tasks — grouping related documents — which makes it a good fit for organizing memories.

Running it via llama.cpp (the local approach for this POC)

Start the llama.cpp server with the embedding flag:

llama-server -hf jinaai/jina-embeddings-v5-text-small-clustering:F16 \
  --embedding --pooling last -ub 32768

Then call the OpenAI-compatible embeddings endpoint:

curl -X POST "http://127.0.0.1:8080/v1/embeddings" \
  -H "Content-Type: application/json" \
  -d '{
    "input": ["Your text to embed goes here"]
  }'

The response contains an embedding array (the vector) for each input string. You store that array as a dense_vector field in Elasticsearch.

Input formatting

The model expects plain text. For this POC's use case (embedding summaries), no special prefix is needed. If you later embed longer documents, prepend "Document: " to the input text, as shown in the HuggingFace examples — the model was trained to distinguish document-type from query-type inputs.

Storing vectors in Elasticsearch

Map the field as dense_vector with dims: 1024 (or your chosen Matryoshka dimension):

"summary_embedding": {
  "type": "dense_vector",
  "dims": 1024,
  "index": true,
  "similarity": "cosine"
}

Then at index time, generate the embedding via llama.cpp and include it in the document body.

Retrieving by similarity (knn search)

POST memory-summary/_search
{
  "knn": {
    "field": "summary_embedding",
    "query_vector": [0.032, -0.14, ...],
    "k": 5,
    "num_candidates": 50
  }
}

query_vector is the embedding of your search text (e.g. the user's first prompt for the new session). The result is the 5 most semantically similar memory summaries.

Matryoshka truncation

This model supports "Matryoshka" embeddings, meaning you can use only the first N dimensions of the vector and still get useful similarity results. For this POC, start with 1024 dims. If storage or query latency becomes a concern, you can truncate to 256 or 512 with only a modest accuracy tradeoff.