Christopher Carroll Smith chriscarrollsmith

Embedding your downloaded Twitter archive data for semantic search and RAG with llm

Getting your tweets

Twitter allows you to download your archive of tweets. You can do this by going to your account settings and requesting your archive. Once you receive the email with the download link, you can download the zip file.

Exploring the data

Top-level keys of Twitter/twitter_archive.json (via jq 'keys' Twitter/twitter_archive.json):

Projects

Runner-Up: Doctor of Credit

Creator: Evan Mullen
Description: Extract structured data about bank deposit bonuses from the Doctor of Credit website.

Prerequisites

Google Chrome CLI entrypoint

Let's hack on `llm`!

If you have uv installed (and you should!), you can install llm globally in a uv-managed tool environment with:

uv tool install llm

If you want to use models other than OpenAI models, you'll need some extensions:

Exploring the Zotero Local Database Schema

On my (Windows) machine, Zotero's local database was located at C:\Users\chris\Zotero\zotero.sqlite. I was able to dump the schema using the sqlite3 command line tool:

sqlite3 zotero.sqlite ".schema" > zotero_schema.sql

Here is the SQL schema dump:

	#!/usr/bin/env bun
	// @bun-dependencies: puppeteer@latest puppeteer-extra@latest puppeteer-extra-plugin-stealth@latest

	import puppeteer from 'puppeteer';

	async function scrapeXTweet(url, metadata) {
	// Launch browser with stealth options
	const browser = await puppeteer.launch({
	headless: true, // or 'new' for new headless mode
	executablePath: '/usr/bin/google-chrome',

	#!/bin/bash

	# create-nl-tool.sh - Programmatically create natural language interfaces
	# for CLI tools
	# Usage: ./create-nl-tool.sh <tool-name>
	# Example: ./create-nl-tool.sh repomix

	set -e

	if [ $# -eq 0 ]; then

	#!/bin/bash

	# Pre-push hook to prevent pushing from the template branch
	# This allows pulling from template remote but prevents accidental pushes

	current_branch=$(git symbolic-ref --short HEAD)

	if [ "$current_branch" = "template" ]; then
	echo "ERROR: Pushing from the 'template' branch is not allowed."
	echo "This branch is reserved for pulling updates from the template repository."

	#!/bin/bash

	# Exit on error
	set -e

	# Load environment variables if .env exists
	if [ -f .env ]; then
	export $(cat .env \| grep -v '^#' \| xargs)
	fi

	from PIL import Image
	import os

	def create_gif(image_folder, gif_name, frame_duration=200):
	"""
	Creates an animated GIF from images in a folder.

	Args:
	image_folder (str): Path to the folder containing the images.
	gif_name (str): Name of the output GIF file.

	from fastapi import FastAPI, Request
	from fastapi.responses import HTMLResponse, StreamingResponse
	import time

	app = FastAPI()

	index_html = """
	<html>
	<head>
	<!-- Include htmx -->