Skip to content

Instantly share code, notes, and snippets.

@eoinsha
Created July 7, 2023 12:58
Show Gist options
  • Save eoinsha/6cf5ce478d36d797d314f161275c5aa6 to your computer and use it in GitHub Desktop.
Save eoinsha/6cf5ce478d36d797d314f161275c5aa6 to your computer and use it in GitHub Desktop.
CloudFormation for CloudTrail Glue Catalog Table
# Working example of a Glue Catalog Data Table for CloudTrail logs that can be used with Athena
CloudTrailGlueTable:
Type: AWS::Glue::Table
Properties:
CatalogId: !Ref AWS::AccountId
DatabaseName: !Ref CloudTrailDatabase
TableInput:
Name: !Sub "${resourcePrefix}_org_cloudtrail"
PartitionKeys:
- Name: account
Type: string
- Name: region
Type: string
- Name: timestamp
Type: string
Parameters:
EXTERNAL: 'TRUE'
projection.enabled: true
# projection.account.type: string
projection.account.type: enum
projection.account.values:
Fn::Join:
- ','
- - Fn::EnumTargetAccounts CloudTrailBinding ${account}
projection.region.type: enum
projection.region.values:
Fn::Join:
- ','
- !Include ../supported-regions.yml
projection.timestamp.format: yyyy/MM/dd
projection.timestamp.interval: 1
projection.timestamp.interval.unit: DAYS
projection.timestamp.range: 2020/01/01,NOW
projection.timestamp.type: date
storage.location.template:
Fn::Join:
- '/'
- - !Sub 's3://${CloudTrailS3Bucket}'
- 'AWSLogs/${account}/CloudTrail/${region}/${timestamp}'
TableType: EXTERNAL_TABLE
StorageDescriptor:
BucketColumns: []
Compressed: false
InputFormat: com.amazon.emr.cloudtrail.CloudTrailInputFormat
Location: !Sub "s3://${CloudTrailS3Bucket}/AWSLogs"
NumberOfBuckets: -1
OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat
SerdeInfo:
Parameters:
serialization.format: '1'
SerializationLibrary: com.amazon.emr.hive.serde.CloudTrailSerde
Columns:
- Name: eventversion
Type: string
- Name: useridentity
Type: struct<type:string,principalId:string,arn:string,accountId:string,invokedBy:string,accessKeyId:string,userName:string,sessionContext:struct<attributes:struct<mfaAuthenticated:string,creationDate:string>,sessionIssuer:struct<type:string,principalId:string,arn:string,accountId:string,userName:string>>>
- Name: eventtime
Type: string
- Name: eventsource
Type: string
- Name: eventname
Type: string
- Name: awsregion
Type: string
- Name: sourceipaddress
Type: string
- Name: useragent
Type: string
- Name: errorcode
Type: string
- Name: errormessage
Type: string
- Name: requestparameters
Type: string
- Name: responseelements
Type: string
- Name: additionaleventdata
Type: string
- Name: requestid
Type: string
- Name: eventid
Type: string
- Name: readonly
Type: string
- Name: resources
Type: array<struct<arn:string,accountId:string,type:string>>
- Name: eventtype
Type: string
- Name: apiversion
Type: string
- Name: recipientaccountid
Type: string
- Name: serviceeventdetails
Type: string
- Name: sharedeventid
Type: string
- Name: vpcendpointid
Type: string
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment