Skip to content

Instantly share code, notes, and snippets.

@farhad-taran
Created January 23, 2024 16:57
Show Gist options
  • Save farhad-taran/5fd6337126d45af2d3b8e0fa423666a5 to your computer and use it in GitHub Desktop.
Save farhad-taran/5fd6337126d45af2d3b8e0fa423666a5 to your computer and use it in GitHub Desktop.
How to use ETags to not refetch already retrieved S3 items

Refetching items from S3 can be quite memory and cpu intensive, and it can be unnecessary if the item has not been changed since. S3 allows for detecting changes to an item through the use of ETags. In the below code I am using the Etag of an item to see if it has changed since the last time it was fetched, if it was not changed since then the cached version of the item is returned instead.

const cache = new Map();
const eTags = new Map();
async function getObjectWithEtags({bucket, key} = {}) {
  const params = {
    Bucket: bucket,
    Key: key,
    IfNoneMatch: eTags.get(key),
  };
  try {
    const res = await exports
      .getClient()
      .getObject(params)
      .promise();
    eTags.set(key, res.ETag);
    cache.set(key, res);
    return res;
  } catch (e) {
    if (e.code == 'NotModified' || e.statusCode == '304') {
      return cache.get(key);
    }
    throw e;
  }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment