Data events were recently enabled for cloudtrail which can show granular detils about modifications on DynamoDB or S3. this can greatly help when trying to audit data access and for security insights. the following terraform script demonstrates how to achieve this:
data "aws_dynamodb_table" "credential-store" {
name = "credential-store"
}
resource "aws_cloudwatch_log_group" "infra-audit-data-access" {
name = "infra-audit-data-access"
We run audit checks as part of our build pipelines, but sometimes it's not possible to fix package audit issues in a timely manner when a critical fix needs to be deployed, and having your build pipeline failing on audit checks and stopping the quick fix deployment is not ideal.
The following is a script which would allow us to skip the audit steps only once if the last commit message contains skip-audit.
if ! git log -1 --stat | grep -q "skip-audit"; then
yarn audit --level high --groups dependencies
exit $(( $? >= 8 ? 1 : 0 ))
fi
const makeCsv = (dataObjectsArray, fieldNamesArray) => {
if (!dataObjectsArray || dataObjectsArray.length == 0) {
return '';
}
const csvLines = []
const formatField = field => `"${field.replace(/"/g, '""')}",`;
csvLines.push(fieldNamesArray.map(formatField).join(''));
dataObjectsArray.forEach(function (dataObj) {
running queries against a dynamodb instance is very hard and non performant if your intention is data analysis.
we can stream all data changes from a dynamodb instance into kinesis,s3,glue crawler and eventually athena. this approach allows us to keep an audit log of all data changes on a dynamoDb and query all these changes using athena.
if however we dont need to keep an audit log of all the changes and we simply need to run queries against the current data in dynamoDb then we can simply connect dynamoDb to aws glue using a glue crawler and athena.
below I will share both options:
we need to mask some specific fields in an object because they are personal identifying info, we would like to do this selectively because in large objects it is possible that nested properties might have the same name but they might not be required to be hidden.
the following function uses recursion to traverse the properties of an object and it accumulates the path to each property, this way the consumer can pass in the paths that it wants hidden and upon detecting these paths it proceeds to hide them.
export default function maskProperties(
obj: Record<string, any>,
pathsToMask: string[],
accumulatedPath = '',
) {
I needed to come up with an in memory cache with a file back up to use in the AWS lambda environment. the storage back up uses the /tmp folder which is shared across the instances of a lambda to store the cache data, this means that the same data can be made available to different invokations and helps with cold starts.
The class, first loads the file, if any data is available it tries to schedule the next refresh, otherwise it will try to refresh the empty data, the refresh function after doing its job uses the finally block to schedule the next refresh.
export type TimestampedData<T> = {
timestamp: string;
content: T;
};
export class FiledMemCache<T> {
I was writing some acceptance tests against a lambda that sent out emails, I thought it would be useful to test the actual snapshot of the HTML thats being sent to the users. since the HTML can be lengthy, I did not want to pollute my test code and I also wanted the HTML snapshot to be readily viewable on a browser as a true representation of the current email layout.
I created a folder which would hold all the snapshot HTML files, and added this index file which would go through all the files in the directoy and export them as strings so that they can be asserted against in the tests.
// Read in the libs from this directory and add them as exports
// This way you can just reference
import * as fs from 'fs';
const snapshots = {};
when doing updates on a document, we usually want to selectively update some fields, or remove other ones. this becomes more complex if you have multiple keys in a document or indexes as the requirements for changing the values of those fields are more strict. the following is a method which gives you the required attributes for doing a dynamic update:
private makeAttributeExpressions = (
fields: Record<string, any>,
documentKeys: string[],
) => {
const exp = {
UpdateExpression: '',
the HTTP sepecification for the PATCH method describes it as a more selective way of updating a resource. this can be easily implemented in typescript or javascript as demonstrated below:
export default function patchObjects(original: any, patch: any) {
const originalCopy = { ...original };
Object.keys(patch).forEach((key) => {
const matchingOriginalProp = originalCopy[key];
if (typeof matchingOriginalProp === 'object') {
originalCopy[key] = patchObjects(matchingOriginalProp, patch[key]);
} else {
some scenarios require us to group objects if a certain property of those objects match. the following js function using the reduce method mimics this behaviour.
function groupBy(list, field) {
const groups = list.reduce((groups, item) => {
const group = (groups[item[field]] || []);
group.push(item);
groups[item[field]] = group;
return groups;
}, []);