- [[Ryan Booz]]: 1 Billion Row Challenge: Comparing Postgres, DuckDB, and Extensions
- Gunnar Morling - 1 Billion Row Challenge https://www.morling.dev/blog/one-billion-row-challenge/
- 1 billion rows is about 13.5GB file
- Big Data -- "data that can't be normally processed given current, 'standard' methods"
- The Five V's: Volume, Variety, Veracity, ...
- More storage options for big data with data lake tech like iceberg and parquet
- Postgres Ingest
- Ingest is the challenge, how quickly can you read in data?
- Copy is single threaded -- can we do multiple copy processes at once? yes, but you need to set things up right for that to work.
- Transaction overhead requires batching
#!/bin/bash | |
# file_info.sh - A script to provide comprehensive information about a file | |
# Usage: ./file_info.sh <filename> | |
set -e | |
# Color definitions | |
RED='\033[0;31m' | |
GREEN='\033[0;32m' |
# config/initializers/validate_constraints.rb | |
Rails.application.config.after_initialize do | |
if Rails.env.development? || Rails.env.test? | |
# Get the check constraint from the database schema | |
connection = ActiveRecord::Base.connection | |
constraint_sql = nil | |
# Try to extract the constraint SQL | |
begin | |
# For PostgreSQL, you can query the information_schema |
#!/bin/bash | |
# Check if fzf is installed | |
if ! command -v fzf &> /dev/null; then | |
echo "Error: fzf is not installed. Please install it first." | |
echo "You can install it with:" | |
echo " - On macOS: brew install fzf" | |
echo " - On Ubuntu/Debian: sudo apt install fzf" | |
echo " - On other systems: see https://github.com/junegunn/fzf#installation" | |
exit 1 |
Some starter prompts for having conversations with AI where the AI asks the questions and you get to answer.
Try pasting one of these into your favorite conversational LLM tool (I like to use Claude Sonnet) and see where their questions take you. Modify them as needed to tailor it to your situation.
I'd like to do a guided form of "Morning Pages" with you, for 15 minutes, where you ask me one question at a time and I answer. The questions can build on each other, but it is also fine to go off in another direction if it could be fruitful. The idea is for me to get all the things in my head, out on paper at the beginning of the day. A light goal is getting some clarify on what to work on today, but the process of "brain dumping" is more important than any particular end result.
Remember, one question at a time. Go ahead and ask me the first question.
/** @type {import("@johnlindquist/kit")} */ | |
// Name: Export all Chrome Tabs | |
// Description: Export all Chrome Tabs to Clipboard | |
let jxa = await npm("@jxa/run"); | |
let result = await jxa.run(() => { | |
// This was taking too long for windows with TONS of Chrome tabs. AppleScript | |
// was timing out. |
[ | |
{ | |
"Plan": { | |
"Node Type": "Sort", | |
"Parallel Aware": false, | |
"Async Capable": false, | |
"Startup Cost": 25.77, | |
"Total Cost": 25.78, | |
"Plan Rows": 1, | |
"Plan Width": 164, |
package main | |
import ( | |
"flag" | |
"fmt" | |
"os" | |
"path/filepath" | |
"regexp" | |
"strings" | |
"time" |
# Send More Money | |
# | |
# Verbal Arithmetic https://en.wikipedia.org/wiki/Verbal_arithmetic | |
letters = %i[s e n d m o r y] | |
check = ->(vals) do | |
# leading zeros are not allowed | |
return false if vals[:s] == 0 || vals[:m] == 0 |