Andrew Cavanagh

Developer.  Project Architect.  Huge Nerd.

(not necessarily in that order)

We use EC2 instances for our webservers. They generally don’t require much disk space, and anything that does require a lot of space we store on s3. However, we often have apps that generate lots of small files, and using a small sized EBS for the webserver has occasionally caused an issue where the disk ran out of inodes. We could have just used a larger EBS, as storage is cheap, but as a solution that seemed both kludgey and boring.

There’s loads of services for monitoring server health, but as we’re using AWS for just about everything and already have an SNS topic that pushes critical alerts into our team slack channel we figured that we might as well take advantage of that and let the apps monitor themselves.

Laravel makes this super easy. I set up a custom command to monitor disk usage - we’re primarily concerned with inode use, so it looks like this:

class CheckInodesAndReportMetric extends Command {

    /**
     * The console command name.
     *
     * @var string
     */
    protected $name = 'inodes';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Check the inode usage and report to Cloudwatch';

    /**
     * Execute the console command.
     *
     * @return void
     */
    public function fire() {
        $percent = exec("df -i |  awk  '$6 == \"/\" { print $5 }'"); //->this grabs the percent of utilized inodes for the root volume.  If you wanted to check space, leave off the '-i'
        $inodes = str_replace('%', '', $percent);
        $this->reportInodesMetric($inodes);

    }

    protected function reportInodesMetric($inodes)
    {
        $namespace = 'SomeCustomNamespaceForMyMetric';
        $name = 'Inode Percent Used';
        $value = 'inode_usage';

        $dimensions = [
            [ 'Name' => 'Custom', 'Value' => $value ],
        ];

        $this->getMetricsClient()->register($namespace, $name, $inodes, $dimensions);
    }

    protected function getMetricsClient(){
        $metricsClient = new MetricsClient();

        return $metricsClient;
    }

}

and the metrics client looks something like this:

class MetricsClient
{
    private $cloudWatchClient;

    public function __construct()
    {
        $config = [
            'credentials' => [
                'key' => env('AWS_ACCESS_KEY'),
                'secret' => env('AWS_SECRET_KEY'),
            ],
            'region'   => env('AWS_REGION'),
            'version'  => 'latest'
        ];

        $this->cloudWatchClient = new CloudWatchClient($config);
    }

    public function register($namespace, $name, $value, $dimensions = [],
        $unit = 'Count')
    {
        $response = $this->cloudWatchClient->putMetricData([
            'Namespace' => $namespace,
            'MetricData' => [
                [
                    'MetricName' => $name,
                    'Timestamp'  => time(),
                    'Value'      => $value,
                    'Unit'       => $unit,
                    'Dimensions' => $dimensions
                ],
            ],
        ]);     
    }
}

This should be enough to push that up to CloudWatch as a custom metric in whatever namespace you defined in the job. We used the built in Laravel scheduler to run that hourly, and set up an alarm that would notify our critical-alert SNS if inode usage crossed 95%.


comments powered by Disqus