Add rotation for log messages in Elastic Beanstalk

If you are using an Elastic Beanstalk server for heavy usage, chances are that your log files, accumulating over time, would end up consuming all the memory in your instance. You would be seeing the following warning

“XX% of root file system is in use, Y MB free”

Chances are that the messages in /var/log/messages of your Linux instance are piling up over time. The solution is log rotation.

Log rotation is essentially a way to compress and store older logs in memory according to set rules. It also discards logs older than a set threshold. For example, your log rotation policy can instruct the instance to compress once a day, and discard logs older than 7 days. This way, the current log file stores logs that are at max 24 hours older. The logs older than 24 hours occupy lower space on the system (because of compression), and the logs older than 7 days are discarded.

Now, elastic beanstalk has a default log rotation policy (using logrotate). You can find it within /etc/logrotate.conf. However, the default policy performs rotation every week, and discards logs after 4 weeks. In some cases, 4 weeks may be too long to overwhelm the system memory.

Therefore, what is required is writing a custom rotation policy for /var/log/messages. Now, before you do that, we need to check where the policy for /var/log/messages is defined. It can be found in /etc/logrotate.d/syslog. If you read that file using sudo cat /etc/logrotate.d/syslog, you’ll see the following contents:

/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
{
  missingok
  sharedscripts
  postrotate
      /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
  endscript
}

What we want to do is remove the handling of the /var/log/messages from this file, and define our custom rules for it. For this, go to your Elastic Beanstalk project folder, and create a folder called .ebextensions (if it does not already exist). Next, create a file within this folder called messages_logrotate.config (you can name it whatever you want). Next, copy paste the following within the file:

files:
  "/etc/logrotate.d/messages":
    mode: "000644"
    owner: root
    group: root
    content: |
      /var/log/messages {
          daily
          rotate 7
          missingok
          compress
          notifempty
          copytruncate
          dateext
          dateformat %s
          olddir /var/log/rotated
      }

  "/etc/logrotate.d/syslog":
    mode: "000644"
    owner: root
    group: root
    content: |
      /var/log/cron
      /var/log/maillog
      /var/log/secure
      /var/log/spooler
      {
        missingok
        sharedscripts
        postrotate
            /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
        endscript
      }     

What we are essentially doing is defining the rotation for /var/log/messages in a new file named ‘messages’ in /etc/logrotate.d/. At the same time, we are removing the handling of /var/log/messages from syslog file. The new rotation policy instructs daily rotation and removal of logs after 7 days. You can get the description of the other parameters here.

Now, there’s one more step. If you directly deploy this file, then the syslog file will be overwritten. However, the older syslog file will be saved as syslog.bak in the same directory. This will cause an error when the logrotation is executing. Therefore, we need to add a postdeploy script to remove the .bak files from the /etc/logrotate.d/ directory.

In order to do that, create a new file called removebak.sh within .platform/hooks/postdeploy in your code repository (you will need to create this path if it doesn’t already exist). Within this file, add the following lines of code:

#!/bin/sh
sudo find /etc/logrotate.d/ -name "*.bak" -exec sudo rm {} \;

This finds all files ending with .bak in the /etc/logrotate.d directory and removes them. Now your code is ready to be deployed.

---

I hope you liked this article. For more articles on IoT in general, check out https://iotespresso.com/