Skip to content

Instantly share code, notes, and snippets.

@katylava
Created December 8, 2010 21:08
Show Gist options
  • Save katylava/733913 to your computer and use it in GitHub Desktop.
Save katylava/733913 to your computer and use it in GitHub Desktop.
split csv file into smaller files
split huge-ass csv file into smaller files
$ wc -l *
40989 IrvineCustomerData.csv
$ split -l 5000 IrvineCustomerData.csv IrvineCustomerData-
$ wc -l *
5000 IrvineCustomerData-aa
5000 IrvineCustomerData-ab
5000 IrvineCustomerData-ac
5000 IrvineCustomerData-ad
5000 IrvineCustomerData-ae
5000 IrvineCustomerData-af
5000 IrvineCustomerData-ag
5000 IrvineCustomerData-ah
989 IrvineCustomerData-ai
40989 IrvineCustomerData.csv
81978 total
$ for f in IrvineCustomerData-*; do head -n 1 IrvineCustomerData.csv > $f.csv && \
cat $f >> $f.csv && rm $f; done
$ wc -l *
5001 IrvineCustomerData-aa.csv
5001 IrvineCustomerData-ab.csv
5001 IrvineCustomerData-ac.csv
5001 IrvineCustomerData-ad.csv
5001 IrvineCustomerData-ae.csv
5001 IrvineCustomerData-af.csv
5001 IrvineCustomerData-ag.csv
5001 IrvineCustomerData-ah.csv
990 IrvineCustomerData-ai.csv
40989 IrvineCustomerData.csv
81987 tota
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment