Last active
June 23, 2026 14:30
-
-
Save stephendolan/367949ff4989d0cd8b2909da4833960e to your computer and use it in GitHub Desktop.
Migrate Tuple's legacy JSONL call recordings into the new SQLite transcript database (index.db). Pure bash + sqlite3, no other deps. Dry-run, confirmations, backups, idempotent.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/usr/bin/env bash | |
| # | |
| # migrate-tuple-jsonl-to-sqlite.sh | |
| # | |
| # Migrate Tuple's legacy JSONL call recordings into the new SQLite transcript | |
| # database (index.db) read by `tuple transcription` and the Tuple MCP server. | |
| # | |
| # Older Tuple builds stored each recording session as a folder under | |
| # "~/Documents/Tuple Calls": | |
| # | |
| # 2026-05-08_13-16-53.852Z@<call-uuid>/ | |
| # events.jsonl one JSON object per line (lifecycle events) | |
| # transcriptions.jsonl one JSON object per line (transcript segments) | |
| # | |
| # Newer builds keep everything in a single SQLite database. This script reads | |
| # the JSONL folders (READ ONLY -- it never modifies them) and inserts the calls, | |
| # recording sessions, transcript segments, events, users, and participants into | |
| # index.db. Full-text search is populated automatically by the database's own | |
| # trigger. | |
| # | |
| # The database must already exist -- the Tuple app creates it and owns its | |
| # schema. Run a transcription with the latest version of the app first; this | |
| # script only adds rows, it never creates the database. | |
| # | |
| # Call titles and summaries are a feature of the new SQLite world, not the JSONL | |
| # recordings, so migrated calls start with empty title/summary. | |
| # | |
| # Multiple session folders that share the same <call-uuid> are grouped into one | |
| # call with multiple recording sessions, exactly like the live app. | |
| # | |
| # Since schema v2, `transcription show` reconstructs each call from the events | |
| # table and surfaces a transcript line only through events.segment_id, so every | |
| # segment needs a paired transcription_finished event or the call shows empty. | |
| # This script writes that pairing for each segment it imports (mirroring the | |
| # app's own migration 002 backfill). | |
| # | |
| # Dependencies: bash and sqlite3 only. Both ship with macOS. No jq, no Python. | |
| # (sqlite3 itself parses the JSON, so transcript text with quotes/commas/emoji | |
| # is handled correctly.) | |
| # | |
| # Safe to re-run: calls already present in the database are skipped, so an | |
| # interrupted run can simply be run again. A re-run also repairs calls left | |
| # empty by an earlier version of this script -- any segment missing its paired | |
| # transcription_finished event is backfilled in place. | |
| # | |
| # Usage: | |
| # ./migrate-tuple-jsonl-to-sqlite.sh [options] | |
| # | |
| # -s, --source DIR Folder of JSONL call recordings (default: the app's | |
| # `transcriptLocation` preference, else | |
| # ~/Documents/Tuple Calls) | |
| # -d, --database FILE Target index.db (default: auto-detected from --env) | |
| # -e, --env ENV prod | staging | dev -- picks the default database path | |
| # (default: prod) | |
| # -n, --dry-run Analyze and report what would happen; write nothing | |
| # -y, --yes Don't ask for confirmation (non-interactive) | |
| # --force Proceed even if Tuple appears to have the database open | |
| # -h, --help Show this help | |
| # | |
| set -euo pipefail | |
| # --------------------------------------------------------------------------- | |
| # Pretty output | |
| # --------------------------------------------------------------------------- | |
| if [ -t 1 ]; then | |
| BOLD=$'\033[1m'; DIM=$'\033[2m'; RED=$'\033[31m'; GREEN=$'\033[32m' | |
| YELLOW=$'\033[33m'; BLUE=$'\033[34m'; RESET=$'\033[0m' | |
| else | |
| BOLD=''; DIM=''; RED=''; GREEN=''; YELLOW=''; BLUE=''; RESET='' | |
| fi | |
| info() { printf '%s\n' "$*"; } | |
| step() { printf '%s==>%s %s\n' "$BLUE$BOLD" "$RESET" "$*"; } | |
| ok() { printf '%s ✓%s %s\n' "$GREEN" "$RESET" "$*"; } | |
| warn() { printf '%s !%s %s\n' "$YELLOW" "$RESET" "$*"; } | |
| die() { printf '%sError:%s %s\n' "$RED$BOLD" "$RESET" "$*" >&2; exit 1; } | |
| usage() { sed -n '2,/^set -euo/p' "$0" | sed '$d; s/^# \{0,1\}//'; exit 0; } | |
| # --------------------------------------------------------------------------- | |
| # Defaults & argument parsing | |
| # --------------------------------------------------------------------------- | |
| SOURCE_DIR="" | |
| DATABASE="" | |
| ENV="prod" | |
| DRY_RUN=0 | |
| ASSUME_YES=0 | |
| FORCE=0 | |
| source_origin="default" # how SOURCE_DIR was chosen (for the report) | |
| while [ $# -gt 0 ]; do | |
| case "$1" in | |
| -s|--source) SOURCE_DIR="${2:?--source needs a value}"; source_origin="flag"; shift 2 ;; | |
| -d|--database) DATABASE="${2:?--database needs a value}"; shift 2 ;; | |
| -e|--env) ENV="${2:?--env needs a value}"; shift 2 ;; | |
| -n|--dry-run) DRY_RUN=1; shift ;; | |
| -y|--yes) ASSUME_YES=1; shift ;; | |
| --force) FORCE=1; shift ;; | |
| -h|--help) usage ;; | |
| *) die "Unknown option: $1 (try --help)" ;; | |
| esac | |
| done | |
| command -v sqlite3 >/dev/null 2>&1 || die "Couldn't find the 'sqlite3' command. | |
| It normally ships with macOS. If you're on a stripped-down system, | |
| install it (e.g. 'brew install sqlite') and re-run." | |
| # macOS bundle id for the chosen environment. Both the default source folder and | |
| # the database path are derived from it. | |
| case "$ENV" in | |
| prod) bundle="app.tuple.app" ;; | |
| staging) bundle="app.tuple.staging" ;; | |
| dev) bundle="app.tuple.dev" ;; | |
| *) die "Unknown --env '$ENV' (expected prod, staging, or dev)." ;; | |
| esac | |
| # Default the source folder from the app's own preference. The transcripts | |
| # folder is user-configurable and Tuple stores it as the `transcriptLocation` | |
| # default; fall back to the historical location if it isn't set. | |
| if [ -z "$SOURCE_DIR" ]; then | |
| if command -v defaults >/dev/null 2>&1; then | |
| SOURCE_DIR="$(defaults read "$bundle" transcriptLocation 2>/dev/null || true)" | |
| fi | |
| if [ -n "$SOURCE_DIR" ]; then | |
| source_origin="transcriptLocation preference" | |
| else | |
| SOURCE_DIR="$HOME/Documents/Tuple Calls" | |
| source_origin="default" | |
| fi | |
| fi | |
| # The daemon keeps its transcript database at index.db in its Application Support | |
| # directory (distinct from the transcripts folder above). There is no preference | |
| # that relocates it, so it's derived purely from the environment. | |
| if [ -z "$DATABASE" ]; then | |
| DATABASE="$HOME/Library/Application Support/$bundle/index.db" | |
| fi | |
| confirm() { # confirm "question" -- returns 0 for yes | |
| [ "$ASSUME_YES" -eq 1 ] && return 0 | |
| local reply | |
| printf '%s%s%s [y/N] ' "$BOLD" "$1" "$RESET" | |
| read -r reply </dev/tty || reply="" | |
| case "$reply" in [yY]|[yY][eE][sS]) return 0 ;; *) return 1 ;; esac | |
| } | |
| # --------------------------------------------------------------------------- | |
| # Validate source and discover calls | |
| # --------------------------------------------------------------------------- | |
| step "Tuple JSONL → SQLite migration" | |
| info " Source folder : ${BOLD}$SOURCE_DIR${RESET} ${DIM}($source_origin)${RESET}" | |
| info " Database : ${BOLD}$DATABASE${RESET} ${DIM}(--env $ENV)${RESET}" | |
| [ "$DRY_RUN" -eq 1 ] && warn "DRY RUN -- nothing will be written." | |
| echo | |
| if [ ! -d "$SOURCE_DIR" ]; then | |
| die "Transcripts folder not found: | |
| $SOURCE_DIR | |
| This should be the folder Tuple saved your call transcripts in. If yours | |
| is somewhere else, point at it with: --source \"/path/to/your/folder\"" | |
| fi | |
| # Collect session folders (those matching *@<uuid> that contain JSONL files) | |
| # and the set of unique call UUIDs, in a temp work area. | |
| WORK="$(mktemp -d "${TMPDIR:-/tmp}/tuple-mig.XXXXXX")" | |
| trap 'rm -rf "$WORK"' EXIT | |
| DIR_LIST="$WORK/dirs.txt" | |
| UUID_LIST="$WORK/uuids.txt" | |
| : >"$DIR_LIST" | |
| shopt -s nullglob | |
| for d in "$SOURCE_DIR"/*@*; do | |
| [ -d "$d" ] || continue | |
| base="$(basename "$d")" | |
| uuid="${base##*@}" | |
| # UUID sanity: hex and dashes only | |
| case "$uuid" in | |
| *[!0-9a-fA-F-]*) continue ;; | |
| esac | |
| [ -f "$d/events.jsonl" ] || [ -f "$d/transcriptions.jsonl" ] || continue | |
| printf '%s\t%s\n' "$uuid" "$d" >>"$DIR_LIST" | |
| done | |
| shopt -u nullglob | |
| if [ ! -s "$DIR_LIST" ]; then | |
| die "No legacy call recordings found in: | |
| $SOURCE_DIR | |
| Expected per-call folders named like '2026-05-08_13-16-53.852Z@<call-id>' | |
| containing events.jsonl / transcriptions.jsonl. Either this isn't the | |
| right folder (point at it with --source) or there's nothing to migrate." | |
| fi | |
| cut -f1 "$DIR_LIST" | sort -u >"$UUID_LIST" | |
| total_dirs=$(wc -l <"$DIR_LIST" | tr -d ' ') | |
| total_calls=$(wc -l <"$UUID_LIST" | tr -d ' ') | |
| # Rough segment/event totals (line counts) for the report. The `|| true` keeps | |
| # `set -o pipefail` from aborting when a folder is missing one of the two files. | |
| src_segments=$({ cat "$SOURCE_DIR"/*@*/transcriptions.jsonl 2>/dev/null || true; } | wc -l | tr -d ' ') | |
| src_events=$({ cat "$SOURCE_DIR"/*@*/events.jsonl 2>/dev/null || true; } | wc -l | tr -d ' ') | |
| ok "Found $total_calls calls across $total_dirs recording-session folders." | |
| info " ~$src_segments transcript segments, ~$src_events events on disk." | |
| echo | |
| # --------------------------------------------------------------------------- | |
| # Inspect target database: existing? open? which calls already migrated? | |
| # --------------------------------------------------------------------------- | |
| # The Tuple app owns creating this database and its schema; this script only | |
| # adds rows. If it isn't there yet, the app hasn't built it -- bail out rather | |
| # than guessing at a schema the app should own. | |
| if [ ! -f "$DATABASE" ]; then | |
| die "Transcript database not found: | |
| $DATABASE | |
| Tuple creates it automatically. Run a transcription with the latest | |
| version of the app to create the SQLite database, then quit Tuple | |
| completely and re-run this script." | |
| fi | |
| # Tuple keeps the database in WAL mode, so an external writer is safe even while | |
| # the app is running -- but the running app may not show the imported calls | |
| # until it re-reads (i.e. until you restart it). Default to asking you to quit | |
| # so results are visible immediately; --force migrates in place anyway. Either | |
| # way the per-call transactions set a busy timeout and wait for the write lock. | |
| if command -v lsof >/dev/null 2>&1 && lsof -- "$DATABASE" >/dev/null 2>&1; then | |
| if [ "$FORCE" -eq 1 ]; then | |
| warn "Tuple appears to be running -- migrating in place (--force)." | |
| warn "Restart Tuple afterward to see the imported calls." | |
| else | |
| die "Tuple appears to be running (it has the database open). | |
| Quit Tuple, then re-run so the imported calls show up right away. | |
| Or pass --force to migrate while it runs (safe; restart to see them)." | |
| fi | |
| fi | |
| # Which of our UUIDs are already in the DB? | |
| sqlite3 "$DATABASE" "SELECT id FROM calls;" 2>/dev/null | sort -u >"$WORK/existing.txt" || : >"$WORK/existing.txt" | |
| already=$(comm -12 "$UUID_LIST" "$WORK/existing.txt" | wc -l | tr -d ' ') | |
| to_migrate=$((total_calls - already)) | |
| ok "Database has $already of $total_calls calls already; $to_migrate to migrate." | |
| # Earlier versions of this script wrote segments without the paired | |
| # transcription_finished events row that schema v2 requires, so those calls | |
| # render empty in `transcription show`. Any segment with no paired event is a | |
| # casualty (live-recorded calls always have the pairing, so they're untouched). | |
| # Count them now -- these all belong to already-migrated calls, since fresh | |
| # calls aren't inserted yet -- and repair them below. This makes a re-run heal | |
| # data that an old run left broken; the per-call insert above keeps new | |
| # migrations correct. | |
| unpaired=$(sqlite3 "$DATABASE" " | |
| SELECT count(*) FROM segments s | |
| WHERE NOT EXISTS (SELECT 1 FROM events e WHERE e.segment_id = s.id);" 2>/dev/null || echo 0) | |
| [ "$unpaired" -gt 0 ] && warn "$unpaired already-migrated transcript segments need repair (empty in 'transcription show')." | |
| echo | |
| if [ "$to_migrate" -eq 0 ] && [ "$unpaired" -eq 0 ]; then | |
| ok "Nothing to do -- every call is already in the database and intact." | |
| exit 0 | |
| fi | |
| # --------------------------------------------------------------------------- | |
| # Dry run stops here | |
| # --------------------------------------------------------------------------- | |
| if [ "$DRY_RUN" -eq 1 ]; then | |
| step "Dry run summary" | |
| [ "$to_migrate" -gt 0 ] && info " Would migrate ${BOLD}$to_migrate${RESET} calls into:" | |
| [ "$unpaired" -gt 0 ] && info " Would repair ${BOLD}$unpaired${RESET} transcript segments from earlier imports into:" | |
| info " $DATABASE" | |
| echo | |
| info "Re-run without --dry-run to perform the migration." | |
| exit 0 | |
| fi | |
| # --------------------------------------------------------------------------- | |
| # Confirm and back up | |
| # --------------------------------------------------------------------------- | |
| if [ "$to_migrate" -gt 0 ] && [ "$unpaired" -gt 0 ]; then | |
| plan="Migrate $to_migrate calls and repair $unpaired transcript segments" | |
| elif [ "$to_migrate" -gt 0 ]; then | |
| plan="Migrate $to_migrate calls" | |
| else | |
| plan="Repair $unpaired transcript segments" | |
| fi | |
| confirm "$plan into $(basename "$DATABASE")?" || { info "Aborted."; exit 0; } | |
| stamp="$(date +%Y%m%d-%H%M%S)" | |
| backup="$DATABASE.bak-$stamp" | |
| cp "$DATABASE" "$backup" | |
| ok "Backed up database to: $backup" | |
| echo | |
| # --------------------------------------------------------------------------- | |
| # Repair earlier imports: backfill the missing transcription_finished events. | |
| # Idempotent (NOT EXISTS guard), so it only touches segments left unpaired by an | |
| # old run and leaves live-recorded calls alone. Mirrors migration 002's backfill. | |
| # --------------------------------------------------------------------------- | |
| if [ "$unpaired" -gt 0 ]; then | |
| step "Repairing $unpaired transcript segments from earlier imports" | |
| if sqlite3 -cmd ".timeout 10000" "$DATABASE" " | |
| INSERT INTO events(recording_session_id, time, category, segment_id, user_id) | |
| SELECT s.recording_session_id, s.start_time, 'transcription_finished', s.id, s.user_id | |
| FROM segments s | |
| WHERE NOT EXISTS (SELECT 1 FROM events e WHERE e.segment_id = s.id);" 2>"$WORK/err.txt"; then | |
| ok "Repaired $unpaired segments (now visible in 'transcription show')." | |
| else | |
| warn "Repair failed:" | |
| sed 's/^/ /' "$WORK/err.txt" >&2 | |
| fi | |
| echo | |
| fi | |
| # --------------------------------------------------------------------------- | |
| # Migrate, one call (transaction) at a time | |
| # --------------------------------------------------------------------------- | |
| migrated=0 | |
| skipped=0 | |
| failed=0 | |
| n=0 | |
| if [ "$to_migrate" -gt 0 ]; then | |
| step "Migrating $to_migrate calls" | |
| while IFS= read -r uuid; do | |
| n=$((n + 1)) | |
| if grep -qxF "$uuid" "$WORK/existing.txt"; then | |
| skipped=$((skipped + 1)) | |
| continue | |
| fi | |
| # All session folders for this call, in chronological order (folder names | |
| # start with an ISO timestamp, so a plain sort is chronological). | |
| # (Built with a read loop rather than mapfile for bash 3.2 / stock macOS.) | |
| dirs=() | |
| while IFS= read -r line; do | |
| dirs+=("$line") | |
| done < <(awk -F'\t' -v u="$uuid" '$1==u {print $2}' "$DIR_LIST" | sort) | |
| script="$WORK/call.sql" | |
| { | |
| echo ".bail on" | |
| echo ".mode ascii" | |
| printf '.separator "\\x1f" "\\n"\n' | |
| echo "CREATE TEMP TABLE IF NOT EXISTS ev_raw(line TEXT);" | |
| echo "CREATE TEMP TABLE IF NOT EXISTS tr_raw(line TEXT);" | |
| echo "PRAGMA foreign_keys=OFF;" | |
| echo "PRAGMA busy_timeout=10000;" # wait up to 10s for the write lock if Tuple is mid-write | |
| echo "BEGIN;" | |
| echo "INSERT OR IGNORE INTO calls(id,summary,title) VALUES('$uuid','','');" | |
| for d in "${dirs[@]}"; do | |
| ev="$d/events.jsonl" | |
| tr="$d/transcriptions.jsonl" | |
| echo "DELETE FROM ev_raw; DELETE FROM tr_raw;" | |
| [ -s "$ev" ] && printf ".import '%s' ev_raw\n" "$ev" | |
| [ -s "$tr" ] && printf ".import '%s' tr_raw\n" "$tr" | |
| # Recording session. Start/end from recording_started/ended events, with | |
| # fallbacks to the earliest/latest event or segment timestamp, then ''. | |
| cat <<SESSION | |
| INSERT INTO recording_sessions(call_id, started_at, ended_at) | |
| VALUES( | |
| '$uuid', | |
| COALESCE( | |
| (SELECT json_extract(line,'\$.time') FROM ev_raw WHERE json_extract(line,'\$.category')='recording_started' LIMIT 1), | |
| (SELECT min(json_extract(line,'\$.time')) FROM ev_raw), | |
| (SELECT min(json_extract(line,'\$.start')) FROM tr_raw), | |
| ''), | |
| COALESCE( | |
| (SELECT json_extract(line,'\$.time') FROM ev_raw WHERE json_extract(line,'\$.category')='recording_ended' LIMIT 1), | |
| (SELECT max(json_extract(line,'\$.time')) FROM ev_raw), | |
| (SELECT max(json_extract(line,'\$.end')) FROM tr_raw), | |
| '') | |
| ); | |
| INSERT OR IGNORE INTO users(id, full_name, short_name, email) | |
| SELECT DISTINCT | |
| json_extract(line,'\$.user.id'), json_extract(line,'\$.user.full_name'), | |
| json_extract(line,'\$.user.short_name'), json_extract(line,'\$.user.email') | |
| FROM ev_raw WHERE json_extract(line,'\$.user.id') IS NOT NULL; | |
| INSERT INTO events(recording_session_id, time, category, user_id, message) | |
| SELECT (SELECT max(id) FROM recording_sessions), | |
| json_extract(line,'\$.time'), json_extract(line,'\$.category'), | |
| json_extract(line,'\$.user.id'), COALESCE(json_extract(line,'\$.message'),'') | |
| FROM ev_raw; | |
| INSERT INTO segments(recording_session_id, user_id, start_time, end_time, text) | |
| SELECT (SELECT max(id) FROM recording_sessions), | |
| json_extract(line,'\$.user_id'), json_extract(line,'\$.start'), | |
| json_extract(line,'\$.end'), COALESCE(json_extract(line,'\$.text'),'') | |
| FROM tr_raw WHERE COALESCE(json_extract(line,'\$.user_id'),0) != 0; | |
| -- Pair each segment with a transcription_finished event. Since schema v2 the | |
| -- unified stream that 'transcription show' reconstructs surfaces a transcript | |
| -- line only through events.segment_id; a bare segment with no paired event is | |
| -- invisible. This mirrors migration 002's backfill, scoped to the session just | |
| -- inserted. burst_id stays NULL (best-effort for old recordings, same as 002). | |
| INSERT INTO events(recording_session_id, time, category, segment_id, user_id) | |
| SELECT recording_session_id, start_time, 'transcription_finished', id, user_id | |
| FROM segments WHERE recording_session_id = (SELECT max(id) FROM recording_sessions); | |
| SESSION | |
| done | |
| # Call-level start/end span all sessions; participants = known users seen. | |
| cat <<FINALIZE | |
| UPDATE calls SET | |
| started_at=(SELECT min(started_at) FROM recording_sessions WHERE call_id='$uuid'), | |
| ended_at =(SELECT max(ended_at) FROM recording_sessions WHERE call_id='$uuid') | |
| WHERE id='$uuid'; | |
| INSERT OR IGNORE INTO participants(call_id, user_id) | |
| SELECT DISTINCT '$uuid', e.user_id | |
| FROM events e JOIN recording_sessions rs ON rs.id=e.recording_session_id | |
| WHERE rs.call_id='$uuid' AND e.user_id IS NOT NULL | |
| AND e.user_id IN (SELECT id FROM users); | |
| FINALIZE | |
| echo "COMMIT;" | |
| } >"$script" | |
| if sqlite3 "$DATABASE" <"$script" 2>"$WORK/err.txt"; then | |
| migrated=$((migrated + 1)) | |
| printf '\r [%d/%d] migrated %s' "$n" "$total_calls" "$uuid" | |
| else | |
| failed=$((failed + 1)) | |
| echo | |
| warn "Failed to migrate $uuid:" | |
| sed 's/^/ /' "$WORK/err.txt" >&2 | |
| if grep -qi "database is locked" "$WORK/err.txt"; then | |
| warn "Tuple held the database too long. Quit Tuple and re-run to finish (already-done calls are skipped)." | |
| fi | |
| fi | |
| done <"$UUID_LIST" | |
| echo; echo | |
| fi | |
| # --------------------------------------------------------------------------- | |
| # Verify and report | |
| # --------------------------------------------------------------------------- | |
| step "Verifying" | |
| # Orphan check limited to what we can observe: events/segments whose session | |
| # is missing (should always be zero). | |
| orphans=$(sqlite3 "$DATABASE" " | |
| SELECT | |
| (SELECT count(*) FROM events e WHERE NOT EXISTS (SELECT 1 FROM recording_sessions r WHERE r.id=e.recording_session_id)) | |
| + (SELECT count(*) FROM segments s WHERE NOT EXISTS (SELECT 1 FROM recording_sessions r WHERE r.id=s.recording_session_id));") | |
| if [ "$orphans" -eq 0 ]; then | |
| ok "Referential integrity OK (no orphaned events or segments)." | |
| else | |
| warn "$orphans orphaned rows detected -- inspect before relying on the data." | |
| fi | |
| # Every segment should now be reachable through a transcription_finished event, | |
| # or it will render empty in 'transcription show'. This must be zero. | |
| still_unpaired=$(sqlite3 "$DATABASE" " | |
| SELECT count(*) FROM segments s | |
| WHERE NOT EXISTS (SELECT 1 FROM events e WHERE e.segment_id = s.id);") | |
| if [ "$still_unpaired" -eq 0 ]; then | |
| ok "Every segment is paired with a transcription_finished event (none will show empty)." | |
| else | |
| warn "$still_unpaired segments still have no transcription_finished event -- they will render empty." | |
| fi | |
| read -r db_calls db_segments db_events <<<"$(sqlite3 -separator ' ' "$DATABASE" \ | |
| "SELECT (SELECT count(*) FROM calls),(SELECT count(*) FROM segments),(SELECT count(*) FROM events);")" | |
| echo | |
| ok "Migrated: ${BOLD}$migrated${RESET} calls" | |
| [ "$unpaired" -gt 0 ] && ok "Repaired: ${BOLD}$unpaired${RESET} segments from earlier imports" | |
| [ "$skipped" -gt 0 ] && info " Skipped: $skipped (already present)" | |
| [ "$failed" -gt 0 ] && warn "Failed: $failed (see messages above; their transactions rolled back)" | |
| echo | |
| info "Database now holds: ${BOLD}$db_calls${RESET} calls, ${BOLD}$db_segments${RESET} segments, ${BOLD}$db_events${RESET} events." | |
| info "Browse with: ${DIM}tuple${ENV:+ --env $ENV} transcription list${RESET}" | |
| echo | |
| ok "Done." | |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This has issues because it was written before the schema updated to our last version.
Here's what I found (with Claude obvi):
Migrated calls show empty in
tuple transcription show(JSONL→SQLite script writes pre-v2 transcript shape)Type: Bug
Area:
migrate-tuple-jsonl-to-sqlite.sh(legacy JSONL →index.db) vs. the v2 "transcription is an event" schema /db::Storeread pathReported: 2026-06-18 (call review with Mikey, Alberto, Eli, Monroe)
Summary
Calls migrated by the JSONL→SQLite shell script render empty in
tuple transcription show <call-id>— the call exists with metadata, andtranscription listshows a segment count, but the transcript body is blank.Live-recorded calls are fine. The segments are physically present in the DB
(search/FTS can even find them), but the unified stream that
showreconstructs comes from the
eventstable, and the script never writes thetranscription_finishedevent rows that point at those segments.Root cause (confirmed in code)
Since schema v2 (
shared/tn/db/migrations/002_transcript_events.h), atranscript line lives in the stream as an events row of category
transcription_finishedwhosesegment_idjoins thesegmentsrow:The read path
query_records(shared/tn/db/Store.cpp:22-71) walks only theeventstable and classifies a row as transcript text iff it has asegment_id(Store.cpp:45), joiningsegmentsone.segment_id. A segmentwith no paired
transcription_finishedevents row is invisible toshow.The migration script inserts segments but never inserts the paired
transcription_finishedevents rows. Its per-session SQL inserts:eventsonly fromevents.jsonl(lifecycle events) — all with NULLsegment_id;segmentsfromtranscriptions.jsonl.There is no third insert linking the two. So every migrated call has segments
but zero
transcription_finishedevents →showreconstructs the stream andfinds no transcript = empty.
Proof this is a known requirement: migration
002does precisely thebackfill the script omits, for exactly this reason
(
002_transcript_events.h:11-18):The script writes data in the pre-v2 shape (segments only) that v2 was
written to repair.
Fix
After inserting a session's segments, insert the paired
transcription_finishedevents rows (mirror 002's backfill, scoped to the session just inserted):
Notes:
burst_idstays NULL — best-effort for old recordings, same as 002.(
query_recordsorders bye.id), so transcript lines clump after eventsrather than interleaving chronologically. Acceptable / matches 002; flag if
exact chronological order in
showis wanted.user_id != 0segment filter is already correct (matches the v3speaker-required constraint) — not the bug.
Re-migration
Already-migrated calls are skipped on re-run (calls present in
callsareskipped), so fixing the script alone won't repair calls already imported — those
need either a one-shot backfill of
transcription_finishedrows for existingmigrated segments, or deleting and re-importing the affected calls.
Related observation (from call)
transcription showfollow mode,transcription_started/droppedmarkersare streamed when they needn't be; they were only needed for the agent JSON
output. Separate cleanup, not this bug.
Screenshot
Pending — attach
tuple transcription show <migrated-id>empty next totranscription listshowing the call with a non-zero segment count.