chatkeeper1.3.0

README-chatkeeper-normalize.md

ChatKeeper Conversation Export Normalizer

chatkeeper 1.3.0 accepts a top-level conversations.json, but newer ChatGPT exports split conversations across files such as conversations-000.json, conversations-001.json, and so on.

If you only care about conversations, the easiest fix is to merge those shards back into a single conversations.json and build a minimal zip containing only that file.

The script for that is:

• /home/paulwright/Downloads/chats/chatkeeper-normalize-conversations.sh

Requirements

• bash
• jq
• zip
• unzip

Usage

From an extracted export directory:

./chatkeeper-normalize-conversations.sh ./not-working-new-format

From the original export zip:

./chatkeeper-normalize-conversations.sh ./not-working-new-format.zip

Write to an explicit output file:

./chatkeeper-normalize-conversations.sh \
  ./not-working-new-format.zip \
  ./not-working-new-format-chatkeeper-conversations.zip

Overwrite an existing output zip:

./chatkeeper-normalize-conversations.sh --force \
  ./not-working-new-format.zip \
  ./not-working-new-format-chatkeeper-conversations.zip

Then run ChatKeeper against the generated zip:

chatkeeper keep ./not-working-new-format-chatkeeper-conversations.zip ./test

What The Script Does

1. Detects whether the input already contains conversations.json or uses sharded conversations-*.json.
2. If sharded, merges the conversation arrays with jq -s 'add'.
3. Builds a minimal zip containing only conversations.json.

This is enough for ChatKeeper's conversation export path. It does not try to preserve non-conversation export data.

Current Limitation

This normalizer fixes the archive-structure problem, but it cannot fix message formats that ChatKeeper itself does not yet understand.

Verified locally against not-working-new-format.zip with chatkeeper 1.3.0:

• 3803 conversations loaded
• 3801 converted
• 2 partially converted because ChatKeeper hit newer Canvas-style message payloads

Those two conversations were:

• JSON Schema for Skewer (670c1087-7928-8010-9742-39e01ee85e85)
• Map Creation Clarification (696fc9c0-333c-832d-83d9-87fdc9d27466)

There was also one non-fatal unhandled tool message reported as super_widget.

chatkeeper-normalize-conversations.sh

#!/usr/bin/env bash

set -euo pipefail
shopt -s nullglob

usage() {
  cat <<'EOF'
Usage:
  chatkeeper-normalize-conversations.sh [--force] INPUT [OUTPUT_ZIP]

Create a minimal ChatKeeper-compatible zip containing only conversations.json.

INPUT can be either:
  - an extracted ChatGPT export directory
  - a ChatGPT export zip file

If OUTPUT_ZIP is omitted, a name is derived from INPUT:
  <input>-chatkeeper-conversations.zip

Options:
  -f, --force   Overwrite OUTPUT_ZIP if it already exists
  -h, --help    Show this help text
EOF
}

die() {
  printf 'Error: %s\n' "$*" >&2
  exit 1
}

need_cmd() {
  command -v "$1" >/dev/null 2>&1 || die "Missing required command: $1"
}

absolute_path() {
  case "$1" in
    /*) printf '%s\n' "$1" ;;
    *) printf '%s/%s\n' "$PWD" "$1" ;;
  esac
}

derive_output_path() {
  local input_path="$1"
  local parent
  local base
  local stem

  if [[ -d "$input_path" ]]; then
    parent=$(dirname "$input_path")
    base=$(basename "$input_path")
    stem="$base"
  else
    parent=$(dirname "$input_path")
    base=$(basename "$input_path")
    stem="${base%.zip}"
  fi

  printf '%s/%s-chatkeeper-conversations.zip\n' "$parent" "$stem"
}

merge_shards() {
  local destination="$1"
  shift
  jq -s 'add' "$@" > "$destination"
}

extract_from_directory() {
  local input_dir="$1"

  if [[ -f "$input_dir/conversations.json" ]]; then
    cp "$input_dir/conversations.json" "$merged_json"
    return
  fi

  local shards=("$input_dir"/conversations-*.json)
  [[ ${#shards[@]} -gt 0 ]] || die "No conversations.json or conversations-*.json found in $input_dir"

  merge_shards "$merged_json" "${shards[@]}"
}

extract_from_zip() {
  local input_zip="$1"
  local -a conversation_json_paths
  local -a shard_paths
  local shard_dir
  local shard_index
  local shard_output

  mapfile -t conversation_json_paths < <(
    unzip -Z1 "$input_zip" | awk '/(^|\/)conversations\.json$/'
  )

  if [[ ${#conversation_json_paths[@]} -gt 1 ]]; then
    die "Found multiple conversations.json files in $input_zip"
  fi

  if [[ ${#conversation_json_paths[@]} -eq 1 ]]; then
    unzip -p "$input_zip" "${conversation_json_paths[0]}" > "$merged_json"
    return
  fi

  mapfile -t shard_paths < <(
    unzip -Z1 "$input_zip" | awk '/(^|\/)conversations-[0-9][0-9][0-9]\.json$/' | sort
  )

  [[ ${#shard_paths[@]} -gt 0 ]] || die "No conversations.json or conversations-*.json found in $input_zip"

  shard_dir="$workdir/shards"
  mkdir -p "$shard_dir"
  shard_index=0

  for shard_path in "${shard_paths[@]}"; do
    shard_index=$((shard_index + 1))
    shard_output=$(printf '%s/%04d.json' "$shard_dir" "$shard_index")
    unzip -p "$input_zip" "$shard_path" > "$shard_output"
  done

  local shards=("$shard_dir"/*.json)
  merge_shards "$merged_json" "${shards[@]}"
}

force=false

while [[ $# -gt 0 ]]; do
  case "$1" in
    -f|--force)
      force=true
      shift
      ;;
    -h|--help)
      usage
      exit 0
      ;;
    --)
      shift
      break
      ;;
    -*)
      die "Unknown option: $1"
      ;;
    *)
      break
      ;;
  esac
done

if [[ $# -lt 1 || $# -gt 2 ]]; then
  usage
  exit 1
fi

need_cmd jq
need_cmd zip
need_cmd unzip

input_path="$1"
[[ -e "$input_path" ]] || die "Input not found: $input_path"

if [[ $# -eq 2 ]]; then
  output_path="$2"
else
  output_path=$(derive_output_path "$input_path")
fi

output_path=$(absolute_path "$output_path")

if [[ -e "$output_path" ]]; then
  if [[ "$force" == true ]]; then
    rm -f "$output_path"
  else
    die "Output already exists: $output_path (use --force to overwrite)"
  fi
fi

workdir=$(mktemp -d /tmp/chatkeeper-normalize-conversations.XXXXXX)
merged_json="$workdir/conversations.json"

cleanup() {
  rm -rf "$workdir"
}

trap cleanup EXIT

if [[ -d "$input_path" ]]; then
  extract_from_directory "$input_path"
elif [[ -f "$input_path" ]]; then
  extract_from_zip "$input_path"
else
  die "Unsupported input: $input_path"
fi

jq -e 'type == "array"' "$merged_json" >/dev/null || die "Merged conversations.json is not a JSON array"
conversation_count=$(jq 'length' "$merged_json")

mkdir -p "$(dirname "$output_path")"

(
  cd "$workdir"
  zip -q "$output_path" conversations.json
)

printf 'Wrote %s\n' "$output_path"
printf 'Conversations: %s\n' "$conversation_count"
printf 'Run next: chatkeeper keep %q OUTPUT_DIR\n' "$output_path"