Easy way to debug and externalize logs from CloudWatch when log events are more

In this section, I will tell you an easy way of how we can debug logs from CloudWatch when the log events are more.

Problem Description: Consider we have a Lambda that processes more than 1000 records and for every processing of the data, an event will be logged in to CloudWatch. Now some of the data in the processing failed. We need to identify how and why the data has failed.

To check the above issue,

1. Login to AWS Console, look for CloudWatch services

2. Get the log group for the respective lambda, (for e.g, the lambda name is hello_world)

3. Look for the log stream and the time stamp for when the data has failed processing by getting into the log group

4. Click on the log stream you will find the log events on how the data got processed.

Issue: Consider our lambda process 1000 records. If we need to identify the particular data for which it has failed, we need to keep on clicking the Load more as shown below until our data shows up which is time-consuming.

In order to avoid this, we can externalize the log stream and look for the data that we need using grep or find option, which will be a very a quick task.

Solution:

1. Login to any of the EC2 box launched in the respective region where the lambda runs through Putty or SSH via Terminal
2. Get the log stream of the log group by using the below command

aws logs describe-log-streams — log-group-name /aws/lambda/hello_world — log-stream-name-prefix 2020/08/17/ — region “us-east-1”

3. Get the log stream name you need based on the prefix passed as mentioned in Step.2 and create a JSON file in the below format and save it as log-events.json

log-events.json

4. Copy the log stream to a log file log-events.log. To do that use the below comment,

aws logs get-log-events — cli-input-json — region “us-east-1” file://log-events.json > log-events.log

Now all the log events, logged by the lambda will be available in the log file. To get the log file to your local desktop location from the EC2 box, use FileZilla or scp command.

Trouble Shooting:

1. When you try to execute the command, “aws logs get-log-events” or “aws logs describe-log-streams”. There can be error thrown saying the user role for the EC2 box in which it is run is not having the required access to get or describe events. To avoid that, for the particular role, Go to AWS IAM -> roles -> select the role -> attach policies

Policies: CloudWatchEventsFullAccess, CloudWatchEventsInvocationAccess