Setting up log rotation for Mesos agents

 ⋅ 2 min read

Setting up log rotation for Mesos agents

Continuing with my series of Mesos posts, I wanted to show how to correctly configure log rotation for your Mesos agent sandboxes. The official documentation on Mesos logging leaves much to be desired in terms of real-world examples, so here's how I did it successfully.

Steps to set up log rotation

  1. Set the --container-logger Agent flag to org_apache_mesos_LogrotateContainerLogger.
  2. Set the --modules Agent flag to file:///path/your-custom-config.json
  3. Save your logrotate to /path/your-custom-config.json.

Example

In my production setup, I prefer using environment variables as they're easier to override and separate from run scripts. For any Mesos flag, you can always use an environment variable of the form MESOS_SOME_FLAG in lieu of a flag of the form --some-flag.

# Default environment variables for Mesos.
# See http://mesos.apache.org/documentation/latest/configuration/master-and-agent/
cat <<EOF | sudo tee /etc/default/mesos
MESOS_CONTAINER_LOGGER=org_apache_mesos_LogrotateContainerLogger
MESOS_MODULES=file:///etc/mesos-agent-settings.json

EOF

Once logrotation is enabled, write your logrotate configuration for both stdout and stderr streams in the modules JSON file. Note that the logrotate directives need to be separated by a \n, as JSON strings do not support newlines.

/etc/mesos-agent-settings.json

The below configuration sets up to keep the last 4 logs each of size up to 25 MB, compresses rotated logs and doesn't email old logs to any address.

# Default log rotation for all agent machines.
cat <<EOF | sudo tee /etc/mesos-agent-settings.json
{
  "libraries": [{
    "file": "/usr/lib/liblogrotate_container_logger.so",
    "modules": [{
      "name": "org_apache_mesos_LogrotateContainerLogger",
      "parameters": [{
        "key": "logrotate_stdout_options",
        "value": "rotate 4\nsize 25M\nmissingok\nnotifempty\ncompress\ndelaycompress\nnomail\n"
      }, {
        "key": "logrotate_stderr_options",
        "value": "rotate 4\nsize 25M\nmissingok\nnotifempty\ncompress\ndelaycompress\nnomail\n"
      }]
    }]
  }]
}

EOF

Mesos
DevOps
Share