Talkamore currently has two separate systems writing journal content:
-
JournalEntry(table:journal_entries) — AI-generated daily roll-ups of chat messages. Created byflush.tswhen the user hits 10 unflushed messages or the 8-hour stale cron fires. One row per user per day. Contains a raw transcript + an AI-generated first-person summary. -
Journal(table:journals) — User-authored documents created in the web frontend's BlockNote editor. Content is opaque JSON ({ title, mode, pages: [...] }). Autosaved every ~1.5s via PUT/api/v1/drafts/:id.
The problem: These two systems overlap. The "remembering" page shows JournalEntry summaries but ignores what the user wrote in Journal. If a user manually journals and also chats with Maya on the same day, they see the AI's rewrite instead of their own writing. Confusing and disrespectful of user effort.
What we're building: A unified JournalBlock model that replaces both tables. Every piece of journal content — whether AI-generated from a chat session or manually written by the user — becomes a "block" on a chronological timeline. A single day can have multiple blocks from different sources, interleaved by time.
Additionally, we're replacing the current flush system with session-based detection: instead of flushing after 10 messages or 8 hours, we detect natural conversation "sessions" (separated by 2+ hours of silence) and generate one summary block per session.
backend/prisma/schema.prisma—Journalmodel (user-written, BlockNote JSON),JournalEntrymodel (AI-generated, one per day),ChatMessagemodel,Usermodelbackend/src/jobs/flush.ts—flushUserBuffer()groups unflushed USER messages by calendar date, concatenates as[HH:MM] messagelines, upserts intojournal_entries, fires summary generation + mood extraction + supermemory ingest.flushStaleBuffers()is hourly cron.FLUSH_THRESHOLD = 10backend/src/lib/chat.ts—persistMessage()saves a ChatMessage,processChat()handles the full chat turn. After each USER message, incrementsunflushedCounton User and callsflushUserBufferwhen threshold is hit (around line 1010-1025)backend/src/lib/summarize.ts—generateJournalSummary()calls GPT-4o to rewrite day's transcript into first-person narrative. Also hasgenerateConversationTitle()backend/src/lib/openai-extract.ts—extractMetadata()extracts mood, energy, themes, moodReason from textbackend/src/lib/crypto.ts—encryptForUser()/decryptForUser()for at-rest encryption with per-user DEKsbackend/src/lib/supermemory.ts—addDocument(),deleteSmDocument(),fireAndLog(),flattenBlockNoteToText()for semantic memory ingestbackend/src/lib/time.ts—toLocalHHMM(),toLocalDateString(),toLocalDateUTC()for timezone-aware formattingbackend/src/lib/usage.ts—recordUsage()for tracking LLM token costsbackend/src/lib/analytics.ts—track()for event analyticsbackend/src/lib/style.ts—extractStyleSignals()for writing style profilingbackend/src/lib/pi-runtime.ts—evictUserSessions()for cache invalidationbackend/src/api/me-routes.ts—handleMeJournal()(GET/api/me/journal) returns entries list,handleMeJournalDay()returns one day,matchMeRoute()dispatches. Response shape:{ user, count, entries: [{ date, summary, content, mood, energy, moodReason, themes, smStatus, createdAt, updatedAt }] }backend/src/api/journals.ts— CRUD for user-written Journal (BlockNote editor): create, get, list as drafts, autosave PUT, rename PATCH, delete. Routes:/api/v1/journals,/api/v1/draftsbackend/src/api/helpers.ts—sendJson(),sendError(),parseJsonBodyOrError()backend/src/lib/auth.ts—requireJwt()for endpoint authbackend/src/index.ts— cron schedules (hourlyflushStaleBuffers)
- ALL text that touches the DB must go through
encryptForUser()/decryptForUser() - Use
toLocalHHMM(),toLocalDateString(),toLocalDateUTC()fromlib/time.ts - Use
fireAndLog()fromlib/supermemory.tsfor fire-and-forget async operations - Use
recordUsage()for ALL LLM calls - Use
track()for analytics events - Use in-memory
Set<string>for per-user/per-session locks (same pattern asflushesInFlightin flush.ts) - Never block the chat response with heavy operations — use
setImmediate()for fire-and-forget
model ChatSession {
id String @id @default(uuid())
userId String
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
startedAt DateTime
endedAt DateTime
closedAt DateTime?
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
messages ChatMessage[]
journalBlock JournalBlock?
@@index([userId, closedAt])
@@index([userId, endedAt(sort: Desc)])
@@map("chat_sessions")
}model JournalBlock {
id String @id @default(uuid())
userId String
user User @relation(fields: [userId], references: [id], onDelete: Cascade)
entryDate DateTime @db.Date
source JournalBlockSource
startedAt DateTime
endedAt DateTime
// for CHAT_SESSION: AI summary as plain text, encrypted
// for USER_WRITTEN / USER_EDITED: BlockNote JSON document, encrypted (as JSON string)
content String
contentLength Int?
// for CHAT_SESSION: the raw "[HH:MM] message" transcript, encrypted
// for USER_WRITTEN: null (the content IS the user's work)
rawTranscript String?
rawTranscriptLength Int?
// first-person narrative summary
// for CHAT_SESSION: AI-generated from the session
// for USER_WRITTEN: optionally AI-generated from user's text (for supermemory/search)
summary String?
summaryLength Int?
mood MoodLevel?
energy EnergyLevel?
moodReason String?
moodReasonLength Int?
themes String[]
// only set for source = CHAT_SESSION
chatSessionId String? @unique
chatSession ChatSession? @relation(fields: [chatSessionId], references: [id])
// the name of the journal (for USER_WRITTEN, maps from old Journal.name)
name String?
smDocumentId String?
smStatus SmStatus @default(PENDING)
createdAt DateTime @default(now())
updatedAt DateTime @updatedAt
// No unique constraint on (userId, entryDate) — multiple blocks per day is the point
@@index([userId, entryDate(sort: Desc)])
@@index([userId, entryDate, startedAt])
@@index([userId, source])
@@map("journal_blocks")
}
enum JournalBlockSource {
CHAT_SESSION
USER_WRITTEN
USER_EDITED
}chatSessionId String?
chatSession ChatSession? @relation(fields: [chatSessionId], references: [id])chatSessions ChatSession[]
journalBlocks JournalBlock[]Keep both old tables intact. They'll be read by the old code paths during the parallel-running period. We'll deprecate them in Phase 5.
npx prisma migrate dev --name add-journal-blocks-and-chat-sessions// session-based journal grouping. replaces the old 10-message / 8h-stale flush.
// a "session" = a window of chat activity. when the user goes silent for
// SESSION_GAP_MS, the session closes and a JournalBlock is created.
import { db } from "./db.js";
import { log } from "./log.js";
import { toLocalHHMM, toLocalDateString, toLocalDateUTC } from "./time.js";
import { extractMetadata } from "./openai-extract.js";
import { generateSessionSummary } from "./summarize.js";
import { addDocument, fireAndLog } from "./supermemory.js";
import { recordUsage } from "./usage.js";
import { track } from "./analytics.js";
import { decryptForUser, encryptForUser } from "./crypto.js";
export const SESSION_GAP_MS = 2 * 60 * 60 * 1000; // 2 hours
const closesInFlight = new Set<string>();Called on every USER message after it's persisted to chat_messages.
Logic:
- Find the most recent open session for this user (
closedAt: null,orderBy: { endedAt: "desc" }) - If found AND
(messageCreatedAt - session.endedAt) < SESSION_GAP_MS:- Update
session.endedAt = messageCreatedAtandchatMessage.chatSessionId = session.idin one transaction - Return session.id
- Update
- If found AND gap >= SESSION_GAP_MS:
- Fire-and-forget
closeSession(openSession.id)viasetImmediate(DO NOT block the chat response) - Fall through to create new session
- Fire-and-forget
- Create new
ChatSessionwithstartedAt = messageCreatedAt, endedAt = messageCreatedAt - Update
chatMessage.chatSessionId = newSession.id - Return newSession.id
This is the core function. It:
- Acquires the in-memory lock (skip if already in
closesInFlight) - Fetches the session with user (timezone, displayName) and messages (role: USER, orderBy createdAt asc)
- If no messages or already closed, skip
- Sets
closedAt = new Date()immediately (prevents re-processing) - Decrypts all message contents
- Builds transcript as
[HH:MM] messagelines - Generates session summary via
generateSessionSummary()(see Phase 3) - Extracts mood/themes via
extractMetadata() - Encrypts summary, moodReason, transcript, content
- Creates a
JournalBlockwith:source: "CHAT_SESSION"entryDate: toLocalDateUTC(session.startedAt, timezone)startedAt: session.startedAtendedAt: session.endedAtcontent: encrypted summary(this is what the user sees)rawTranscript: encrypted transcript(the original [HH:MM] lines)summary: encrypted summary(same as content for chat sessions)- mood, energy, themes, moodReason
chatSessionId: sessionId
- ALSO upserts into the old
JournalEntrytable (for backwards compat during migration):- Same logic as
flush.ts flushGroup— append transcript to existing entry or create new one - This ensures the old frontend code and old API endpoints still work
- Same logic as
- Marks all USER messages in the session as
flushed: true - Fire-and-forget: supermemory ingest with
customId: journal_block_${block.id} - Fire-and-forget: style profile refresh (same as flush.ts)
- Analytics:
track("session_closed", { message_count, duration_minutes, ... }) - Records usage for summary + extract LLM calls
Run every 15 minutes via node-cron.
- Find all ChatSessions where
closedAt: nullANDendedAt < (now - SESSION_GAP_MS) - For each, call
closeSession(id)with try/catch
Add generateSessionSummary() alongside the existing generateJournalSummary(). Same structure, same MODEL (gpt-4o), same error handling.
System prompt:
Rewrite a chat session transcript as a first-person journal entry, written in the user's own voice. Use "I" statements. Preserve their actual words and phrasing where possible, don't over-literary or clean up their grammar. Keep it honest, not preachy or over-reflective.
Capture the ARC of the session — how the conversation started, what shifted, where it landed. If multiple topics came up, show how they connected (they usually do emotionally, even when they seem unrelated).
Pull 1-2 direct quotes from the transcript that capture the most raw/honest moments. Wrap them naturally into the prose.
Target 80-200 words depending on session length. A 5-minute session might only need 2 sentences. A 2-hour session might need a full paragraph.
Plain prose, no bullet points, no headers, no timestamps (the frontend renders the time range separately). Never mention an AI assistant, chatbot, Maya, Sage, or Luna — write as if this is the user's own journal.
User message format:
NAME: {displayName}
DATE: {dateStr}
SESSION: {startTime} – {endTime}
CHAT TRANSCRIPT:
{transcript}
First-person summary:
Temperature 0.3, max_tokens 400. Apply stripEmDashes(). Build usage via buildExtractUsage().
Function signature:
export interface GenerateSessionSummaryInput {
displayName: string;
dateStr: string;
startTime: string;
endTime: string;
transcript: string;
}
export async function generateSessionSummary(opts: GenerateSessionSummaryInput): Promise<SummaryResult>Returns all journal blocks (chat sessions + user-written) grouped by date, ordered chronologically within each day.
Query params: ?limit=90 (max days to return, default 90)
Response shape:
{
"user": { "id": "...", "displayName": "...", "timezone": "..." },
"days": [
{
"date": "2026-05-13",
"blocks": [
{
"id": "uuid",
"source": "CHAT_SESSION",
"startedAt": "2026-05-13T14:00:00Z",
"endedAt": "2026-05-13T16:15:00Z",
"startTime": "2:00 PM",
"endTime": "4:15 PM",
"content": "Work was brutal today...",
"summary": "Work was brutal today...",
"mood": "LOW",
"energy": "MEDIUM",
"moodReason": "deadline pressure",
"themes": ["work stress"],
"name": null,
"createdAt": "...",
"updatedAt": "..."
},
{
"id": "uuid",
"source": "USER_WRITTEN",
"startedAt": "2026-05-13T20:30:00Z",
"endedAt": "2026-05-13T20:45:00Z",
"startTime": "8:30 PM",
"endTime": "8:45 PM",
"content": { "title": {...}, "mode": "manual", "pages": [...] },
"summary": null,
"mood": null,
"energy": null,
"moodReason": null,
"themes": [],
"name": "balcony moment",
"createdAt": "...",
"updatedAt": "..."
}
]
}
]
}Implementation notes:
- Fetch JournalBlocks for the user, ordered by
entryDate desc, thenstartedAt ascwithin each date - Group by
entryDatein application code - Decrypt
content,summary,moodReasonper block - For
CHAT_SESSIONblocks:contentis plain text (the summary) - For
USER_WRITTEN/USER_EDITEDblocks:contentis the BlockNote JSON. Parse the encrypted string back to JSON before sending. Content was stored asJSON.stringify(blockNoteDoc)then encrypted. - Format
startTime/endTimeusingtoLocaleTimeString("en-US", { hour: "numeric", minute: "2-digit", timeZone: user.timezone }) - Limit: count distinct dates, stop at
limitdays
Returns all blocks for a specific date. Same block shape as above, just for one day.
Autosave for user-edited blocks. Used when:
- User creates a new manual entry (source: USER_WRITTEN)
- User edits a chat session block (source flips: CHAT_SESSION → USER_EDITED)
Body: { content: { title, mode, pages } } (same BlockNote shape as current drafts PUT)
Logic:
- requireJwt, owner-check
- Fetch the block
- If block.source === "CHAT_SESSION", flip to "USER_EDITED"
- Encrypt
JSON.stringify(body.content)and update the block - Fire-and-forget: supermemory re-ingest (same debounce pattern as journals.ts — 2-min window)
- Return updated block
Create a new user-written block.
Body: { name: string, content?: { title, mode, pages } }
Logic:
- requireJwt
- Create JournalBlock with:
source: "USER_WRITTEN"entryDate: toLocalDateUTC(now, user.timezone)startedAt: nowendedAt: nowname: body.namecontent: encrypted default or provided content
- Return created block
Same pattern as journal delete in journals.ts. Owner-scoped, SM cleanup fire-and-forget.
DO NOT remove or change these during Phase 4:
- GET
/api/me/journal(reads fromjournal_entries) - GET
/api/me/journal/:date(reads fromjournal_entries) - POST/GET/PUT/PATCH/DELETE
/api/v1/journalsand/api/v1/drafts(reads fromjournals)
They continue serving the old data while the frontend migrates to the new endpoints. The dual-write in closeSession() (Phase 2) ensures journal_entries stays populated.
Add the new routes to the server dispatcher:
/api/me/journal-blocks→ handleMeJournalBlocks/api/me/journal-blocks/:date→ handleMeJournalBlocksDay/api/v1/journal-blocks→ handleJournalBlockCreate (POST)/api/v1/journal-blocks/:id→ handleJournalBlockPut (PUT), handleJournalBlockDelete (DELETE)
After the USER message is persisted (around line 738 where persistMessage is called), add:
import { assignMessageToSession } from "./sessions.js";
// After: const userMessageId = await persistMessage(...)
// Add:
assignMessageToSession(user.id, userMessageId, new Date()).catch((err) => {
log.error("session assignment failed", {
userId: user.id,
messageId: userMessageId,
err: err instanceof Error ? err.message : String(err),
});
});The existing unflushedCount increment + flushUserBuffer threshold at lines 1010-1025 stays in place. Both systems run in parallel:
- Sessions create JournalBlocks (new system)
- Flush creates JournalEntries (old system, for backwards compat)
Both mark messages as flushed: true, but they won't conflict because:
- Session detection runs on every message (assigns to session, potentially closes old session)
- Flush runs on threshold (10 messages) or stale cron (8h)
- The
flushed: trueflag is set by whichever runs first; the other sees no unflushed messages and no-ops
import { closeStaleSessionsCron } from "./lib/sessions.js";
cron.schedule("*/15 * * * *", () => {
closeStaleSessionsCron().catch((err) =>
log.error("stale session cron failed", {
err: err instanceof Error ? err.message : String(err),
}),
);
});Run ONCE after all code is deployed. Backfills existing data into JournalBlock.
From journal_entries → JournalBlock (source: CHAT_SESSION):
- For each JournalEntry:
- Decrypt content and summary
- Create JournalBlock with source CHAT_SESSION
entryDate= JournalEntry.entryDatestartedAt= JournalEntry.createdAt (or derive from first [HH:MM] in content)endedAt= JournalEntry.updatedAt (or derive from last [HH:MM] in content)content= re-encrypt the summary (or content if no summary)rawTranscript= re-encrypt the raw contentsummary= re-encrypt the summary- Copy mood, energy, moodReason, themes, smDocumentId, smStatus
- DO NOT delete the JournalEntry row
From journals → JournalBlock (source: USER_WRITTEN):
- For each Journal:
entryDate= toLocalDateUTC(journal.createdAt, user.timezone)startedAt= journal.createdAtendedAt= journal.updatedAtcontent= encryptJSON.stringify(journal.content)(the BlockNote JSON)name= journal.name- Copy smDocumentId, smStatus
- DO NOT delete the Journal row
Safety measures:
- Process in batches of 50 users
- Skip users who already have JournalBlock rows (idempotent re-runs)
- Log progress every 10 users
- Dry-run flag that counts what would be migrated without writing
After the frontend is fully switched to the new endpoints and the migration script has run:
- Remove the dual-write to
journal_entriesfromcloseSession() - Remove old flush logic from
chat.ts(unflushedCount increment + flushUserBuffer call) - Remove
flushStaleBufferscron - Remove
unflushedCountfrom User model - Mark old API endpoints as deprecated (add deprecation header), then remove after 30 days
- Drop
journal_entriesandjournalstables after 30 days of new system running clean
| File | Action | Phase |
|---|---|---|
backend/prisma/schema.prisma |
Add ChatSession, JournalBlock, JournalBlockSource enum, update ChatMessage and User |
1 |
backend/src/lib/sessions.ts |
NEW — assignMessageToSession, closeSession, closeStaleSessionsCron |
2 |
backend/src/lib/summarize.ts |
Add generateSessionSummary |
3 |
backend/src/api/journal-blocks.ts |
NEW — CRUD endpoints for JournalBlock | 4 |
backend/src/api/me-routes.ts |
Add handleMeJournalBlocks, handleMeJournalBlocksDay, update matchMeRoute |
4 |
backend/src/api/server.ts |
Add route dispatch for new endpoints | 4 |
backend/src/lib/chat.ts |
Wire assignMessageToSession after USER message persist |
5 |
backend/src/index.ts |
Add 15-min cron for closeStaleSessionsCron |
5 |
backend/scripts/migrate-to-journal-blocks.ts |
NEW — one-time backfill script | 6 |
| Parameter | Default | Location |
|---|---|---|
SESSION_GAP_MS |
2 hours | sessions.ts |
| Cron interval | 15 min | index.ts |
| Summary max_tokens | 400 | summarize.ts |
| Summary temperature | 0.3 | summarize.ts |
| SM ingest debounce | 2 min | journal-blocks.ts |
- Never drop old tables until Phase 7 (30+ days after full migration)
- Dual-write to both JournalEntry (old) and JournalBlock (new) during transition
- Never block the chat response — session assignment and closing are fire-and-forget
- All text encrypted — every string that touches the DB goes through encryptForUser/decryptForUser
- Migration script is idempotent — safe to re-run, skips users who already have JournalBlock rows
- Old API endpoints stay untouched — no changes to
/api/me/journal,/api/v1/journals,/api/v1/drafts