Last active
September 28, 2018 15:14
-
-
Save biancalpadilla/44df44b1a95a9ef0d71d9b786295c954 to your computer and use it in GitHub Desktop.
CSV too big? Split it using this script...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/bin/bash | |
# To run, cd into the directory (make sure the file you want to split is named file.csv) type command `sh csv-splitter.sh` | |
# You can convert any txt file into a csv file by changing the extension to .csv | |
# Split your csv (or txt) file at a specific line number by chaning the 10000 in line 17 to any value you'd like. | |
# Look for a file called file.csv | |
FILE=$(ls -1 | grep file.csv) | |
NAME=${FILE%%.csv} | |
# Save the first line in the file as your header | |
head -1 $FILE > header.csv | |
# Save the rest of file (line 2 and on) as your data file | |
tail -n +2 $FILE > data.csv | |
# Split data at every X number of lines | |
split -l 10000 data.csv | |
# Iterate over each split file | |
for a in x?? | |
do | |
# Add the header to each new split file | |
cat header.csv $a > $NAME.$a.csv | |
done | |
rm data.csv x?? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment