- Official docs: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-lake.html
- Guide: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-lake.html
- Pricing: https://aws.amazon.com/cloudtrail/pricing/
- Login/assume a role that has sufficient permissions to create a LakeTrail and interact with Organizations* (See below)
- Open up the CloudTrail service and Select Lake on the left side (https://console.aws.amazon.com/cloudtrailv2/home?region=us-east-1#/lake#/lake/eventDataStores)
Step 1:
- Click 'Create event data store'
- Enter the name for the datastore - ex: LakeTrail-Organization-ManagementEvents
- Set the retention period - the default is 7 years (2555 days)
- If you only want the trail active for your current region (doesn't really follow AWS best practices, but), check "Include only the current region"
- You will need to flip to every region and enable the Lake manually if that's how you want to segment things
- If you want to enable for your organization, select "Enable for all accounts ..."
- Add any tags you need, and click Next
Step 2:
- Select the types of events you want to capture - the defaults are sane
- Click Next
Step 3:
- Review your configuration, and if you're happy, click Create event data store
This will take some time - the status will be 'Creation in progress' for a bit. Grab a sip of coffee.
One the state is 'Enabled' everything is setup. You will need to wait a bit though - just like regular CloudTrail, the service takes many minutes to deliver logs, and it does not import existing Trail data.
The query editor is ok - I wish they'd just used Athena, it's more functional and follows a more standard UI design
In the editor, you can start with something simple just to see how much data you have:
select count(*) from <insert ugly data store ID>
select * from <why isn't this just the name of the event store?>
Note - when you hit Run, you will see the 'Command Output' table populate with the summary of the run. To get the actual results you need to click on the Query Resuts tab which is to the left (why) of the Command Output tab.
You can also use the one of the pre-canned queries by clicking on the Sample queries tab, and clicking on the Query SQL (why not the query name?)
This will create a new tab in the editor view, populated with a query that should get you going.
Unfortunately they didn't build the example to cleverly use todays date so it would just work, you need to update it, so don't forget or you'll be querying in the past - which gets you a Successful query, even though you likely don't have data from 2021-07-22 (default datetime).
- Really cool feature is the Results History - you can see all the queries run, and their results, so you don't have to run the same query again if you forgot to snag the output, or want to see what something returned yesterday vs today
- Login as
I haven't been able to figure out everything, but at least:
- ListEventDataStores
- GetEventDataStore
- GetQueryResults
- StartQuery
- DescribeQuery
- CreateEventDataStore
and some Organizations stuff for sure, but I haven't figured that out yet.
need to figure out more, as the default 'ReadOnly' role doesn't have sufficient permissions to run queries.
-
CloudTrail Lake does not create a trail in the CloudTrail service
- This means that Config rules that validate that CloudTrail is enabled may not work
- CloudTrail Insights will not work
- The normal CloudTrail UI will not work for investigations
-
When you delete an Event Store, it sits in Pending Deletion status for some time, and you can do Actions -> Restore on it
-
CloudTrail Lake does not create an S3 bucket you an interact with in any way
-
There doesn't appear to be any way to see how much data you're storing
-
Unlike Athena, you cannot click on the field names to populate them in your query, but at least they're easy to copy/paste
-
Would be nice if there was a way to have a button that would update date fields to match todays date, or the ability to do relative dates or a variable for $todays_date so that a saved query just works
-
I can't find a way to validate that everything is enabled properly in organization accounts
- Looking in Cloudtrail I can't find events in accounts indicating it's turned on?
select count(*), recipientAccountId from <id> group by recipientAccountId
to see all the accounts that sent CT data
-
Clikcing the cog in the results and trying to turn columns on/off doesn't work
-
No way to export data to CSV
-
You can only have 5 Query tabs open in the UI - this is too little.
-
Seems like saved searches are not shared between users - stored as cookies? How do we build a library of useful searches for people to use? Having a Tab of 'Common Searches' we can populate with examples would be huge
-
how can I make scheduled queries of this data to do something like reporting? (ie: run daily to find out which services hit throttling to help us find problems)
-
status: failed on jobs. it would be nice if there was more to why