Skip to content

Instantly share code, notes, and snippets.

@cengizhancaliskan

cengizhancaliskan/pandas read csv.py

Created November 21, 2019 13:51
Show Gist options
  • Save cengizhancaliskan/e2aff781378793454b7615cb02745fc7 to your computer and use it in GitHub Desktop.
Save cengizhancaliskan/e2aff781378793454b7615cb02745fc7 to your computer and use it in GitHub Desktop.
Download ZIP
Pandas read csv by with chunk for big size CSV files
Raw
pandas read csv.py
filename = 'filename.csv'
chunksize = 10 ** 6
chunk_list = []
for chunk in pd.read_csv(filename, chunksize=chunksize, error_bad_lines=False):
chunk_list.append(chunk)
df = pd.concat(chunk_list)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment