How to rotate logs in AWS CloudWatch?

cloudwatch log rotation
cloudwatch logs
cloudwatch agent
cloudwatch log format
cloudwatch logs pricing
cloudwatch apache logs
timestamp is more than 2 hours in future.

I am using the Amazon CloudWatch Log Agent (AWS documentation) to upload logs from my EC2 instances to the CloudWatch console.

For my CloudWatch Log Agent configuration on EC2, I have the following:

state_file = /var/awslogs/state/agent-state  

file = /var/www/html/logs/applog.log
log_group_name = MyApp
log_stream_name = applog.log
datetime_format = %Y-%m-%d %H:%M:%S

My question is, how do I get the applog.log to rotate on a daily basis? In the AWS Documentation (link above), it mentions being able to configure log rotation policies, but I can't find any mention/example of how to actually do this. I've tried specifying the log_stream_name as applog_%Y-%m-%d.log, but it interprets this literally.

Any ideas or pointers in the right direction would be very welcome - thanks!

I don't think there is a way to rotate the log_stream_name using Amazon's CloudWatch Log Agent. The log rotation described in the documentation is related to ingesting log files that get rotated by your system, the CloudWatch Log Agent does not perform any log rotation itself.

According to the documentation the only variables allowed in the log_stream_name property are {instance_id}, {hostname} and {ip_address}

How to rotate logs in AWS CloudWatch?, in question:, click on "Never expire", in the "Expire Events After" column. The popup will allow you to choose the retention period. One or more log files are created every five minutes in the specified bucket. Even when logs are published directly to an S3 bucket, CloudWatch Logs charges apply. For more information, see Deliver Logs to S3 on Amazon CloudWatch Pricing.

I think what you are asking for is log expiry. At least that is what I was looking for. And here is how you can expire logs after a certain amount of time:

  1. Through sam template
  2. Through console:
    1. On the AWS console dashboard, navigate to Cloudwatch>>Logs.
    2. In the table which lists all the logs, for your particular log in question:, click on "Never expire", in the "Expire Events After" column. The popup will allow you to choose the retention period

CloudWatch Logs Agent Reference, CloudWatch Logs Agent FAQs. What kinds of file rotations are supported? The following file rotation mechanisms are supported: Renaming existing log files with  With CloudWatch Logs, you can perform real-time analysis of the log data, store the data in highly durable storage, and manage the data with the CloudWatch Logs Agent. AWS retains log data published to CloudWatch Logs for an indefinite time period unless you specify a retention period.

According to the Agent documentation, the log filename can have wildcards.

Cloudwatch Agent Docs

File can point to a specific file or multiple files (using wildcards such as /var/log/system.log*). Only the latest file is pushed to CloudWatch Logs based on file modification time.

So, you can just start writing to a new file that matches your pattern and everything should be fine.

Amazon CloudWatch Logs, The CloudWatch Logs agent makes it easy to quickly send both rotated and non- rotated log data off of a host and into the log service. You can then access the  Add Metrics Filter. Go to CloudWatch Logs. Select a Log groups radio button (don’t click on the log group!) Select the Create Metric Filter button. Put in your pattern on the field. Click Next. Use a good namespace and name for your metric. Click Advanced on metric details. Here you can select to

Customize Log Files in Elastic Beanstalk, Rotate your logs; (Optional) Stream your logs to CloudWatch. Note: If you have a custom log file or if one of your logs is missing from the  Create an IAM role for Cloudwatch. To set up AWS custom logs, first, you need to create and add an IAM role to your instance. This IAM role will have write access to Cloudwatch service so that all the logs can be shipped to Cloudwatch.

Troubleshoot Pushing Log Data to CloudWatch, Why can't I push log data to CloudWatch Logs with the awslogs agent? If logs stopped pushing after a log rotation, check the supported log  cdk-log-notifier: Filter CloudWatch logs and post to Slack. The AWS CDK Construct to build a system that gather CloudWatch logs, filter and post to Slack. Example Usage. Watch the all logs contains "ERROR" from Lambda functions.

Manually Create or Edit the CloudWatch Agent Configuration File , Explains how to manually create the CloudWatch agent configuration file, including an IAM role to use when sending metrics and logs to a different AWS account. The agent keeps the rotated log files for up to seven days, and it keeps as  Make dashboard-related API calls through the console for free rather than making them through the AWS CLI or an SDK. CloudWatch Logs. Charges are incurred by ingestion and storage of Amazon CloudWatch Logs. Check the IncomingBytes metric to determine ingested data amounts. Refer to your AWS bill to determine data storage amounts.