Skip to content

Instantly share code, notes, and snippets.

@revolutionisme

revolutionisme/s3_to_pandas.py

Created March 16, 2020 15:30
Show Gist options
  • Save revolutionisme/1d7603789cfbdc02f0d47a05137b24ee to your computer and use it in GitHub Desktop.
Save revolutionisme/1d7603789cfbdc02f0d47a05137b24ee to your computer and use it in GitHub Desktop.
Download ZIP
Reading and Writing data to/from s3 using pandas and s3fs
Raw
s3_to_pandas.py
import pandas as pd
import s3fs
fs = s3fs.S3FileSystem(profile_name='aws_profile')
with fs.open("s3://my-bucket/file.csv", "rb") as f:
csv_df = pd.read_csv(f)
fs = s3fs.S3FileSystem(profile_name='other_aws_profile')
with fs.open("s3://another-bucket/file.csv", "wb") as f
df.to_csv(f)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment