What is AWS CloudTrail?

AWS CloudTrail is an auditing, compliance monitoring, and governance tool from Amazon Web Services (AWS). It’s classed as a “Management and Governance” toola in the AWS console.

With CloudTrail, AWS account owners can ensure every API call made to every resource in their AWS account is recorded and written to a log. An API call can be made:

• when a resource is accessed from the AWS console
• when someone runs an AWS CLI command
• when a REST API call is made to an AWS resource

These actions can be coming from:

• Human users (e.g. when someone spins-up an EC2 instance from the console)
• Applications (e.g. when a bash script calls an AWS CLI command)
• another AWS service (e.g. when a Lambda function writes to an S3 bucket)

CloudTrail saves the API events in a secured, immutable format which can be used for later analysis.

In this article, we will learn the basics of AWS CloudTrail, see how to create and enable custom trails, see where the trail logs are saved, and how to analyze CloudTrail logs. We will also compare CloudTrail with another AWS service: CloudWatch Logs.

Why AWS CloudTrail?

Someone working in DevSecOps can view, search for, or analyze CloudTrail logs to find:

• any particular action that happened in the account
• the time the action happened
• the user or process that initiated the action
• the resource(s) that were affected by the action

Having this kind of visibility can be useful for post security breach reviews, proactive AWS monitoring for account vulnerabilities or ensuring adherence to compliance standards. What’s more, events from custom trail events can be used to trigger specific actions.

Is AWS CloudTrail Enabled By Default?

AWS CloudTrail is now enabled for all users by default.

AWS CloudTrail Features

Amazon CloudTrail has a number of features you would expect from a monitoring and governance tool. These features include:

• AWS CloudTrail is “Always On,” enabling you to view data from the most recent 90 days.
• Event History to allow you to see all changes made.
• Multi-region configuration.
• Log file integrity validation and encryption.
• Data events, management events, and CloudTrail Insights.

[Learn More: AWS Config vs CloudTrail]

Amazon CloudTrail Pricing

Amazon CloudTrail pricing is free of charge if you set up a single trail to deliver a single copy of management events in each region. With CloudTrail, you can even download, filter, and view data from the most recent 90 days for all management events at no cost.

Keep in mind Amazon S3 charges will apply based on your usage.

Additionally, you can use AWS CloudTrail Insights by enabling Insights events in your trails. AWS CloudTrail Insights are charged per the number of events in each region. Pricing is as follows:

• Management Events: $2.00 per 100,000 events
• Data Events: $0.10 per 100,000 events
• CloudTrail Insights: $0.35 per 100,000 write management events

CloudTrail Event History

AWS account administrators don’t have to do anything to enable CloudTrail: it’s enabled by default when an account is created. This is the default trail. Information in this trail is kept for the last 90 days in a rolling fashion.

To view the default trail, we can open the CloudTrail console, and choose “Event history” from the navigation pane:

From the event history, it’s possible to choose a date range, a particular resource type or resource name, event name, AWS access key ID and other filters to narrow down the search. The image below shows how we are narrowing down our search to view events related to RDS DBInstance type resource.

Clicking on a particular event’s record will show more information. The snippet below shows part of the JSON record for the DeleteDBInstance event:

{
 "eventVersion": "1.05",
 "userIdentity": {
 "type": "IAMUser",
 "principalId": "XXXXXXXXXXXXXXXXXXXX",
 "arn": "arn:aws:iam::1234567890:user/Administrator",
 "accountId": "1234567890",
 "accessKeyId": "XXXXXXXXXXXXXXXX",
 "userName": "Administrator",
 "sessionContext": {
 "attributes": {
 "mfaAuthenticated": "false",
 "creationDate": "2019-03-22T03:03:44Z"
 }
 },
 "invokedBy": "signin.amazonaws.com"
 },
 "eventTime": "2019-03-22T03:04:34Z",
 "eventSource": "rds.amazonaws.com",
 "eventName": "DeleteDBInstance",
 "awsRegion": "us-east-1",
 "sourceIPAddress": "12.34.56.78",
 "userAgent": "signin.amazonaws.com",
 "requestParameters": {
 "dBInstanceIdentifier": "mysql-db",
 "skipFinalSnapshot": true,
 "deleteAutomatedBackups": true
 },
...

We can see an IAM user named “Administrator” deleted the “mysql-db” instance at a particular date and time in the US-East-1 region and did not make a snapshot before deleting the instance. Also, the user could log in to the console without Multi-Factor Authentication (MFA).

Information like this can help highlight the need for users to start using MFA, or for administrators to start reviewing access to RDS.

Creating a Trail

It’s also possible to create custom trails. A trail is a user-created audit definition that can capture one or more types of events. Unlike Event history, CloudTrail trail logs are not limited to 90 days retention. They can be delivered to an S3 bucket or to AWS CloudWatch Logs and configured to send SNS notifications when a particular event happens.

In the image below, we can see a trail called “Trail1”. The trail’s log files are delivered to an S3 bucket called “athena-cloudtrails”

To define a trail, a number of parameters need to be defined. The images below show each of these parameters.

First, we need to give our trail a name. This can be anything meaningful. We also need to specify if the trail will audit account activities in all regions or only the current region. For multi-account setups, we can also specify if the trail should be enabled at the Organization level.

Next, we specify if we want to track “read events” like “Describe” or “List” API operations, “write events” like “Create” or “Delete” operations, both type of operations or none at all. These events are considered “management events” because they are related to actions performed on the resources, not actions happening within the resources.

Next, we have “data events”. Data events track API operations happening within a specific resource. At the time of writing, two types of resource operations are supported: S3 and Lambda.

With data events, it’s possible to track S3 bucket level operations such as “PutObject”. We can track it for all buckets, or specific buckets:

For Lambda, the trail can capture an event every time any function or a specific function is invoked:

Finally, we need to specify where will be the trail data stored:

We can create a new bucket or choose an existing bucket for storing the trail log files. The files will be saved under a folder in the S3 bucket. The folder structure has the following naming style:

/AWSLogs/<account-id>/CloudTrail/<region>/yyyy/mm/dd/. 

We can choose to put a file name prefix as well.

There are two options to ensure the logs are secured: each file with a KMS key, and optionally, let CloudTrail validate the file. With validation, CloudTrail checks if the generated log file was altered before it was saved to S3.

Administrators can also choose to get notified via SNS alert every time a CloudTrail log file is created.

AWS CloudTrail Files

The image below shows files from the trail we created before (Trail1), saved under the athena-cloudtrails bucket. The files are gzipped archives and have a naming pattern of:

<account_id>_CloudTrail_<region>_yyyymmdd<time>_<unique_hex_number>.json.gz

If validation is enabled, CloudTrail creates another separate folder structure with the naming pattern:

/AWSLogs/<account_id>/CloudTrail-Digest/<rergion>/yyyy/mm/dd

These folders are used to store the digest files. A digest file stores the names of the log files delivered to the S3 bucket in last one hour, their hash values and a “digital signature” of the last digest file. CloudTrail uses the digital signatures and the hash values to validate that files have not been tampered with since they were stored.

Enabling / Disabling Trails

We can enable or disable an existing trail by toggling the button at the top right corner of the trail’s property screen:

This may be necessary when you have a number of trails running in your account and you are not sure which one is used by your log management monitoring platform. You can selectively disable one trail after another and check.

Sending AWS CloudTrails to CloudWatch

Like CloudTrail, Amazon CloudWatch is a core service of the AWS platform. Simply put, CloudWatch can be considered as the eyes and ears of any AWS account. While CloudTrail tracks and records API calls made to objects, CloudWatch offers a number of facilities for monitoring other resources in the account, sending alerts based on the resource state, scheduling Lambda functions and other jobs, and hosting log files from different AWS services and resources.

Some typical use cases of CloudWatch can be:

• AWS billing alerts
• EC2 auto-scaling alerts
• Performance monitoring dashboards for RDS instances
• CloudWatch events for Lambda functions or data pipelines
• CloudWatch log groups and log streams for different resource types (e.g. VPC flow logs)

It’s possible to publish CloudTrail events to CloudWatch Logs. With this approach, CloudTrail can take advantage of CloudWatch features such as:

• Easy searching through logs
• Sending alerts on logged events
• Creating metric filters and dashboards from logged events

To send a trail to CloudWatch, two things need to exist:

• A CloudWatch log group and a log stream
• An IAM role and its policy that grants CloudTrail access to CloudWatch

The feature to send CloudTrail events to CloudWatch Logs becomes available after the trail has been created. To use CloudWatch, the trail needs to be edited afterward. In the image below, we are editing and configuring our Trail1 to publish its logs to CloudWatch.

Clicking the “Configure” button brings up the next section:

CloudWatch needs to know the log group and log stream names where the events will be published to. We can choose the default log group (“CloudTrail”) as suggested, or specify an existing log group. Each trail configured to send its logs to CloudWatch will have a log stream under this log group.

When we click “Continue”, the wizards asks for the IAM role to use:

Once again, we can specify our own custom role and policy, or let the wizard create a role and policy for us. The default name for the role will be “CloudTrail_CloudWatchLogs_Role”. The IAM policy attached to this role will grant CloudTrail “CreateLogStream”, and “PutLogEvents” rights to CloudWatch Logs.

Once the IAM role and policy have been specified, we can save the changes.

Next, we can browse to the log group and see the log stream for the trail has been created:

The naming pattern for the log stream is:

<account_id>_CloudTrail_<region>

Inside the log stream, we can view the log events:

Although it’s simple to send CloudTrail logs to CloudWatch, users need to be mindful of a limitation. The maximum event size in CloudWatch Logs is 256 KB. CloudTrail respects that and will not send an event’s data to CloudWatch Logs if it’s more than that size.

CloudTrail Best Practices

Here are some best practice tips for using CloudTrail:

• If you are using AWS Organizations to manage your multi-account setup, create a trail to apply to the organization. This trail will be created in the master account and will log events from the master account and other member accounts. The trail will serve as a “master” log for all activities in your AWS setup.
  
• Include all regions in your Trail. You may think you are creating AWS resources in only one region, but chances are, someone may create a resource in a different region. This is typically the case for PoCs or when people don’t pay attention. If not careful, these resources could be running undetected, costing money.
  
• Unless you are interested in finding out who is accessing or listing your resources, record only “write events” in the trail. This will show resource creation, modification and deletion events, and reduce log size.
  
• People often create multiple trails. Too many logs can increase S3 storage costs over time and cause confusion. Create one trail to capture everything unless there is a need to create separate trails.
  Also, don’t create separate trails in each region, choose a “master region” and configure your trail to capture events from all regions. The reason to include all regions in your trail is to ensure it’s visible from every regions’ CloudTrail console. By default, a trail that does not include all regions is only visible in the region where it was created.
  
• If you are searching for recent events (in the last 90 days), use event history instead of searching through S3 files. This will improve the query time.
  
• Don’t send CloudTrail logs to CloudWatch unless you are sure none of your event data will ever cross the 256 KB limit. This is even more important if you are using CloudWatch alarms for security and network-related API calls. If some events are dropped because of their sizes, you may miss important alerts.
  
• Instead of manually searching through the logs, use industry-standard log management and SIEM tools like Sumo Logic. We will talk about how Sumo Logic can provide valuable insights from your CloudTrail logs in a future article.

Final Words: AWS CloudTrail

This was a high-level introduction to AWS CloudTrail. As we saw, it’s easy to set up. Once running, the logs can be a valuable source of information for forensic analysis and governance audit. Setting up the trails is the first step for the DevSecOp. The next step would be searching through the logs and analyzing those for insight. For a shortcut, click here to learn more about the Sumo Logic CloudTrail monitoring tools.

Additional Resources

• How to Read, Search, and Analyze AWS CloudTrail Logs