Skip to content

Instantly share code, notes, and snippets.

@gromgit
Last active February 19, 2025 10:14
Show Gist options
  • Save gromgit/6965be955d504d53023439efaff196fe to your computer and use it in GitHub Desktop.
Save gromgit/6965be955d504d53023439efaff196fe to your computer and use it in GitHub Desktop.
#!/usr/bin/env bash
cmd() {
echo ">>> $(printf '%q ' "$@")" >&2
time "$@"
}
sqlite=/opt/homebrew/opt/sqlite/bin/sqlite3
csvs=(perfect_0.csv perfect_1.csv imperfect_2.csv)
# Generate CSVs
for i in $(seq -f %07.0f 9999999); do
echo "${i},This,is,line,${i},of,a,nonsense,CSV,that,exists,just,to,debunk,the,assumption,that,missing,columns,cause,performance,collapse,in,SQLite,${RANDOM}"
done > perfect_0.csv
# 2nd CSV with new PKs
sed -e 's/^/1/' < perfect_0.csv > perfect_1.csv
# 3rd CSV with new PKs and missing last column
sed -e 's/^/2/; s/,[^,]*$//' < perfect_0.csv > imperfect_2.csv
for i in "${csvs[@]}"; do
echo "===== ${i} ($(wc -l < "${i}") rows) ====="
head -5 "$i"
done
rm -f test.db
$sqlite test.db <<EOSQL
CREATE TABLE t(a INTEGER PRIMARY KEY, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p, q, r, s, t, u, v, w, x, y INTEGER);
CREATE INDEX xtra_1 ON t(a,b,c,f,k,l,q,m,g,u);
CREATE INDEX xtra_2 ON t(c,f,k,l);
EOSQL
for i in "${csvs[@]}"; do
cmd bash -c "$sqlite test.db '.import --csv ${i} t' 2>/dev/null"
done
@gromgit
Copy link
Author

gromgit commented Feb 17, 2025

Conclusion: performance collapse reported by OP is NOT due to missing column.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment