Skip to content

Instantly share code, notes, and snippets.

@adeubank
Created August 14, 2014 23:49
Show Gist options
  • Save adeubank/df475c4426cd34e74171 to your computer and use it in GitHub Desktop.
Save adeubank/df475c4426cd34e74171 to your computer and use it in GitHub Desktop.
Bash script that connects to a mysql DB to dump column value. It performs this in batches instead of one try. Useful when exporting large amount of data.
#!/bin/bash
# Connects to a MySQL database to dump data in batches.
# Great for dumping large datasets.
LIMIT=1000000
OFFSET=0
MAX_ROWS=1000000000
# This loops until it has read MAX_ROWS
while [ $OFFSET -lt $MAX_ROWS ]
do
echo "Fetching rows starting $OFFSET"
echo "SELECT COL_NAME FROM TABLE_NAME LIMIT $LIMIT OFFSET $OFFSET" | mysql DATABASE_NAME --skip-column-names -B -u MYSQLUSERNAME -pMYSQL_PASSWD >> output.csv
OFFSET=$((OFFSET+LIMIT))
done
# All you have to do is tweak the LIMIT and MAX variable to your liking.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment