Binoy Shah ShahBinoy
ShahBinoy
/ pandas_s3_streaming.py
Created
November 1, 2020 05:51
— forked from uhho/pandas_s3_streaming.py
Streaming pandas DataFrame to/from S3 with on-the-fly processing and GZIP compression
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| def s3_to_pandas(client, bucket, key, header=None): | |
| # get key using boto3 client | |
| obj = client.get_object(Bucket=bucket, Key=key) | |
| gz = gzip.GzipFile(fileobj=obj['Body']) | |
| # load stream directly to DF | |
| return pd.read_csv(gz, header=header, dtype=str) | |
| def s3_to_pandas_with_processing(client, bucket, key, header=None): |
ShahBinoy
/ etc-init.d-cassandra
Last active
August 29, 2015 14:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| #!/bin/bash | |
| # init script for Cassandra. | |
| # chkconfig: 2345 90 10 | |
| # description: Cassandra | |
| # script slightly modified from | |
| # http://blog.milford.io/2010/06/installing-apache-cassandra-on-centos/ | |
| . /etc/rc.d/init.d/functions | |