openclaw-dreaming-guide.md

Dreaming Guide: Background Memory Consolidation

A practical guide to understanding, enabling, and tuning OpenClaw's dreaming system — the automatic background process that turns short-term memory signals into durable long-term knowledge.

Why Dreaming Exists

OpenClaw agents accumulate memory throughout the day: daily notes, session transcripts, recall traces from searches. Most of this material is useful in the moment but doesn't belong in long-term storage. Without a consolidation step, you face one of two bad outcomes:

• Too aggressive: every fleeting detail lands in MEMORY.md, bloating it with noise.
• Too conservative: nothing ever gets promoted, and genuinely important patterns are lost.

Dreaming solves this with a three-phase background sweep that scores short-term signals over time and only promotes the ones that cross evidence thresholds. Think of it as a curatorial pipeline: ingest, reflect, then carefully promote.

Dreaming is opt-in and disabled by default.

How It Works

When enabled, memory-core creates a managed cron job (default: 3 AM daily) that runs a full dreaming sweep. Each sweep executes three phases in sequence:

Phase 1: Light Sleep (Sort and Stage)

Light phase is the ingestion layer. It:

1. Reads recent daily memory files (memory/YYYY-MM-DD.md) and parses them into snippet chunks.
2. Ingests session transcripts into per-day corpus files under memory/.dreams/session-corpus/.
3. Deduplicates entries using Jaccard similarity (threshold 0.9).
4. Stages candidates in the short-term recall store.
5. Records "light phase signal" hits — these boost ranking in the deep phase later.
6. Writes a ## Light Sleep block into the daily memory file (when storage mode includes inline output).
7. Optionally generates a dream diary narrative entry.

Light phase never writes to MEMORY.md. It only stages and records signals.

Phase 2: REM Sleep (Reflect and Extract Patterns)

REM phase looks for recurring themes across the staged material. It:

1. Reads all short-term recall entries within the REM lookback window (default: 7 days).
2. Extracts recurring themes by analyzing concept tag frequency.
3. Identifies "candidate truths" — entries that show up repeatedly with high confidence.
4. Writes a ## REM Sleep block with reflections.
5. Records REM signal hits (these also boost deep ranking).
6. Generates a dream diary narrative entry.

REM phase never writes to MEMORY.md either. It produces reflective signals that inform the deep phase.

Phase 3: Deep Sleep (Promote to Long-Term Memory)

This is where promotion actually happens. Deep phase:

1. Takes all candidates from the short-term recall store.
2. Scores each one using six weighted signals (see ranking table below).
3. Applies phase reinforcement boosts from light and REM signal hits.
4. Filters out candidates that don't pass the threshold gates.
5. Rehydrates surviving snippets from live daily files (so deleted or stale content is skipped).
6. Appends promoted entries to MEMORY.md under a dated ## Promoted From Short-Term Memory section.
7. Writes a deep sleep report and generates a dream diary narrative entry.

Deep phase is the only phase that writes to MEMORY.md.

Deep Ranking Signals

Signal	Weight	What it measures
Relevance	0.30	Average retrieval quality across all recalls
Frequency	0.24	Total number of short-term signals accumulated
Query diversity	0.15	How many distinct query contexts surfaced the entry
Recency	0.15	Time-decayed freshness (14-day half-life)
Consolidation	0.10	Multi-day recurrence strength
Conceptual richness	0.06	Concept-tag density from snippet and path

Light and REM phase hits add a small recency-decayed boost (up to 0.05 and 0.08 respectively) on top of the base score.

Threshold Gates

A candidate must pass all three gates to be promoted:

Gate	Default	Meaning
minScore	0.8	Weighted composite score must be at least this high
minRecallCount	3	Entry must have been recalled at least this many times
minUniqueQueries	3	Entry must have surfaced from at least this many distinct queries

These gates prevent one-off mentions from being promoted. A memory must demonstrate sustained, diverse relevance.

The Dream Diary

Alongside the machine-readable state, dreaming produces a human-readable Dream Diary in DREAMS.md. After each phase with enough material, a background subagent generates a short, creative narrative entry (80-180 words) written from the perspective of "a curious, gentle, slightly whimsical mind reflecting on the day."

The diary is visible in the Gateway Dreams tab and is intended for human browsing only — it is not a promotion source.

Where Things Live on Disk

Machine state (memory/.dreams/)

File	Purpose
short-term-recall.json	All tracked recall entries and their scores
phase-signals.json	Light/REM hit counts per entry key
daily-ingestion.json	Daily file change tracking
session-ingestion.json	Session file change tracking
session-corpus/YYYY-MM-DD.txt	Ingested session message snippets
short-term-promotion.lock	File lock during promotion
events.jsonl	Audit log of dreaming events

Human-readable output

File	Purpose
DREAMS.md	Dream Diary with ## Light Sleep, ## REM Sleep, ## Deep Sleep blocks
memory/dreaming/deep/YYYY-MM-DD.md	Optional separate deep phase reports
MEMORY.md	Long-term memory where promoted entries land

Getting Started

Enable Dreaming

The fastest way is the slash command in any channel:

/dreaming on

Or add it to your config file:

{
  "plugins": {
    "entries": {
      "memory-core": {
        "config": {
          "dreaming": {
            "enabled": true
          }
        }
      }
    }
  }
}

Change the Sweep Schedule

Default is 3 AM daily. To run every 6 hours instead:

{
  "plugins": {
    "entries": {
      "memory-core": {
        "config": {
          "dreaming": {
            "enabled": true,
            "frequency": "0 */6 * * *"
          }
        }
      }
    }
  }
}

Check Status

/dreaming status

Or via CLI:

openclaw memory status --deep

Disable Dreaming

/dreaming off

Manual and Debugging Workflows

Preview Promotions Without Applying

See what would be promoted if you ran a deep sweep now:

openclaw memory promote

Apply Promotions Manually

Run a deep promotion and write results to MEMORY.md:

openclaw memory promote --apply

Limit to the top 5 candidates:

openclaw memory promote --apply --limit 5

Explain Why Something Would or Wouldn't Promote

Useful for tuning thresholds or understanding the scoring:

openclaw memory promote-explain "router vlan"
openclaw memory promote-explain "router vlan" --json

Preview REM Reflections

See what REM phase would produce without writing anything:

openclaw memory rem-harness
openclaw memory rem-harness --json

Configuration Reference

All settings live under plugins.entries.memory-core.config.dreaming.

Key	Default	Description
enabled	false	Master switch
frequency	"0 3 * * *"	Cron schedule for full sweeps
timezone	(agent default)	Timezone for day boundary calculations
verboseLogging	false	Detailed candidate logging
storage.mode	"inline"	"inline", "separate", or "both"
storage.separateReports	false	Write per-phase report files
phases.light.limit	100	Max candidates to process in light phase
phases.light.lookbackDays	2	How far back light reads daily files
phases.deep.limit	10	Max promotions per sweep
phases.deep.minScore	0.8	Minimum weighted score to promote
phases.deep.minRecallCount	3	Minimum recall signals required
phases.deep.minUniqueQueries	3	Minimum distinct query contexts required
phases.deep.recencyHalfLifeDays	14	Recency decay half-life in days
phases.deep.maxAgeDays	30	Maximum candidate age in days
phases.rem.lookbackDays	7	How far back REM reads recall entries
phases.rem.limit	10	Max REM candidates per sweep
phases.rem.minPatternStrength	0.75	Minimum pattern strength for REM themes

Tuning Guide

Too Many Promotions

If MEMORY.md is growing too fast:

1. Raise phases.deep.minScore (try 0.85 or 0.9).
2. Raise phases.deep.minRecallCount (try 5).
3. Lower phases.deep.limit (try 5).
4. Shorten phases.deep.maxAgeDays so older candidates expire sooner.

Too Few Promotions

If nothing is getting promoted and you're losing important context:

1. Lower phases.deep.minScore (try 0.7).
2. Lower phases.deep.minRecallCount to 2.
3. Increase phases.deep.limit to allow more per sweep.
4. Extend phases.deep.maxAgeDays to give candidates more time to accumulate signals.

Sweep Frequency

• Daily (default): good for most users. Low resource usage, steady promotion.
• Every 6 hours: for active agents with high daily memory throughput.
• Weekly (0 3 * * 0): for agents that don't accumulate much short-term memory.

Debugging Candidate Scoring

1. Enable verboseLogging: true to see per-candidate scores in the event log.
2. Use openclaw memory promote-explain "<query>" to inspect a specific candidate.
3. Check memory/.dreams/events.jsonl for detailed phase execution logs.

How Dreaming Integrates with the Rest of OpenClaw

Daily notes + Sessions + Recall traces
        │
        ▼
  ┌─────────────┐
  │ Light Phase │  Ingest, dedupe, stage, record signals
  └──────┬──────┘
         │
         ▼
  ┌─────────────┐
  │  REM Phase  │  Extract themes, record reinforcement signals
  └──────┬──────┘
         │
         ▼
  ┌─────────────┐
  │ Deep Phase  │  Score, threshold, promote → MEMORY.md
  └─────────────┘
         │
         ▼
   Dream Diary (DREAMS.md) — human-readable narrative only

• Memory search (openclaw memory search) feeds short-term recall signals into the promotion pipeline during normal agent operation.
• Daily memory files (memory/YYYY-MM-DD.md) are the primary source material for light phase ingestion.
• Session transcripts (~/.openclaw/agents/<id>/sessions/*.jsonl) are the secondary source.
• Gateway startup reconciles the managed cron job, so config changes take effect on next gateway restart.
• The Dreams UI tab in the Gateway shows live status, phase counts, and the dream diary.

Related

• Dreaming concept doc
• Memory
• Memory Search
• memory CLI
• Memory configuration reference