Skip to content

Instantly share code, notes, and snippets.

View jflasher's full-sized avatar

Joe Flasher jflasher

View GitHub Profile
@jflasher
jflasher / index.js
Created July 2, 2020 12:03
Lambda function to make an S3 Invetory object publicly accessible
console.log('Loading function');
const aws = require('aws-sdk');
const s3 = new aws.S3({ apiVersion: '2006-03-01' });
exports.handler = (event, context, callback) => {
// Get the object from the event and show its content type
const bucket = event.Records[0].s3.bucket.name;
const key = decodeURIComponent(event.Records[0].s3.object.key.replace(/\+/g, ' '));
const params = {
@jflasher
jflasher / intelligent-tiering.md
Created February 1, 2019 15:21
Instructions for setting up a lifecycle policy for S3 Intelligent-Tiering

Amazon S3 now supports a new storage class called Intelligent Tiering. This will be a very attractive option for many customers who have datasets that are accessed in unpredictable patterns. You can set this storage class when uploading any new data. However, the below instructions will allow you to set up a lifecycle policy that will change the storage class of data that already exists in your bucket.

To set up a lifecycle policy to change the storage class of the data currently stored in your Amazon S3 bucket, follow the below steps.

  1. Visit the S3 console and go to your bucket of interest.

  2. Click on the Management tab at the top and select + Add lifecycle rule.

  3. Enter a rule name of your choice (e.g., Convert to Intelligent Tiering storage class). Unless you want to filter which data is converted to the new storage class, you can leave the prefix/tag filter field

@jflasher
jflasher / details.md
Created February 23, 2018 14:56
Accessing data older than 90 days from OpenAQ

Currently, only data within the last 90 days is available via the OpenAQ API. However, there is much more data available on OpenAQ and a variety of different access mechanisms. Note also that there is work under way to bring back to the API a mechanism to access the data older than 90 days, details here.

If you're looking to query across all the data or even easily export the data (or a subset of it), the easiest way to do that currently is using a service like Amazon Athena. I'll provide some directions on how to do that below, but at a high level, this will let you make any query of the entire dataset that you'd like (written in SQL). I'll also provide some sample queries so you can see what's possible.

On to the directions!

  1. You will need to create an AWS account if you don't currently have one, you can start this process at htt