Created
August 10, 2015 08:04
-
-
Save MarkEdmondson1234/c119bfe81af5d5ab81c8 to your computer and use it in GitHub Desktop.
combine all csv files in a folder
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| ## assuming you have a folder full of .csv's to merge | |
| ## csv's must all have identical column names. | |
| folder <- "./path/to/csv/files" | |
| filenames <- list.files(folder) | |
| all_files <- Reduce(rbind, lapply(filenames, read.csv)) |
If it’s that big I think you need to use the right tool for the job such as BigQuery to load the files into a database. If not that then I would use command line tools such as awk and head to manipulate the files directly.
…________________________________
From: Ramom Ferreira <[email protected]>
Sent: Tuesday, July 30, 2019 5:15 PM
To: MarkEdmondson1234
Cc: Mark; Author
Subject: Re: MarkEdmondson1234/combineCSVfiles.R
I have a folder full of .csv't to merge, but they have different column names and may have different order of columns. Also, they sum up to 33.5gb so I can't load all of them in the memory. Do you know any way I can combine all the csv files in the folder?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://gist.github.com/c119bfe81af5d5ab81c8?email_source=notifications&email_token=AAYCPLAFVAETXFFBKVHKTWTQCBLJ3A5CNFSM4IH5PDL2YY3PNVWWK3TUL52HS4DFVNDWS43UINXW23LFNZ2KUY3PNVWWK3TUL5UWJTQAFWGI4#gistcomment-2985102>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAYCPLA2344C62ZSL53HKADQCBLJ3ANCNFSM4IH5PDLQ>.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I have a folder full of .csv's to merge, but they have different column names and may have different order of columns. Also, they sum up to 33.5gb so I can't load all of them in the memory. Do you know any way I can combine all the csv files in the folder?