We have been on Ec2 for almost two years but recently, we needed to monitor a few stuffs on server so we did a bit of experiences on that and deployed it a couple of months back. We found that we had a pretty good result so I am going to share it here as usual.
To monitor the following things on Amazon Ec2 virtual machine and send the alerts based on some conditions.
- CPU Utilization
- Memory Available
- Disk Space Available
- VM Status
Methods of monitoring
There are a few ways to monitor the instances of EC2 but we chose to use Amazon CloudWatch and New Relic.
- Cloud Watch
- New Relic
Amazon Cloud Watch
The reason is simple. We chose it because Amazon provides it. There is no cost for monitoring some basic stuffs (Amazon CloudWatch Pricing) so it fits us well.
Basic Monitoring metrics (at five-minute frequency) for Amazon EC2 instances are free of charge, as are all metrics for Amazon EBS volumes, Elastic Load Balancers, and Amazon RDS DB instances.
New and existing customers also receive 10 metrics (applicable to Detailed Monitoring for Amazon EC2 instances or Custom Metrics), 10 alarms, and 1 million API requests each month at no additional charge.
Amazon EC2 Console – Default Monitoring
This is what you will see when you open up the EC2 console. You can enable the monitoring and create the alarms in this console as well.
CloudWatch – Monitoring
Once you have enabled the default monitoring in EC2 console, you will see the following metrics in CloudWatch console.
Of course, this is not what we need. We need to monitor the disk space, memory utilization and etc. So we need to create some custom scripts for that.
AWS SDK for .NET
“AWS SDK for .NET” allows you to create the scripts that can create the custom metric on CloudWatch. You can download the SDK from this link ( http://aws.amazon.com/sdkfornet/). We want to monitor the disk space and memory which doesn’t include in EC2 default monitoring so we need to install this SDK and create the script for that.
Amazon CloudWatch Monitoring Scripts for Microsoft Windows Server
You can roll your own script for monitoring what you want to monitor but it’s always a good idea to google before creating your own because if your requirements is very common then someone might be already created the script for that.
I found a brunch of monitoring scripts from this link ( http://aws.amazon.com/code/7932034889155460 ). This package includes the scripts for the following metrics.
- Memory Utilization (%)
- Memory Used (MB)
- Memory Available (MB)
- Page File Utilization (%)
- Page File used (MB)
- Page File available (MB)
- Disk Space Utilization (%)
- Disk Space Used (GB)
- Disk Space Available (GB)
- Perfmon Counters.
Amazon Access Key ID and Secret Access Key
To run those scripts, you will probably need to provide the access key and it wasn’t that straight forward to find your own secret access key on Amazon so I captured the screenshots of navigation.
Bugs in Amazon CloudWatch Monitoring Scripts
You will get the following error when you try to load the Amazon powershell module.
Error Message in text (for Google. of course! )
Copyright (C) 2012 Microsoft Corporation. All rights reserved.
Import-Module : The specified module ‘C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell.psd1’ was not loaded
because no valid module file was found in any module directory.
At C:\Users\michael.sync\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1:1 char:1
+ Import-Module “C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell.psd1”
+ CategoryInfo : ResourceUnavailable: (C:\Program File…PowerShell.psd1:String) [Import-Module], FileNot
+ FullyQualifiedErrorId : Modules_ModuleNotFound,Microsoft.PowerShell.Commands.ImportModuleCommand
This issue occurs because of the wrong path in powershell profile. Look the screenshot below for the file name and its location and you can fix the path.
Note: $env:psmodulePath is the automatic variable which holds the path used to discover modules. If it’s not set, PowerShell looks in c:\windows\system32\WindowsPowerShell\1.0\modules and MyDocuments\WindowsPowerShell\modules
After fixing the wrong, you should be able to run any script from |Amazon monitoring scripts”.
if you can manage to run the scripts that you requires the metrics to be appeared then you can trigger those scripts from windows scheduler every 5 mins or so.
And then you will see new custom metrics as below in CloudWatch dashboard. You can go ahead and create some alerts on new custom metric.
Here is the scripts that I am using for monitoring the memory utilization and disk space.
.\mon-put-metrics-mem.ps1 -aws_credential_file C:\Users\michael.sync\Downloads\AmazonCloudWatchMonitoringWindows\awscreds.conf -mem_util -mem_used -mem_avail -page_avail -page_used -page_util -memory_units Megabytes .\mon-put-metrics-disk.ps1 -aws_credential_file C:\Users\michael.sync\Downloads\AmazonCloudWatchMonitoringWindows\awscreds.conf -disk_drive C:, D: -disk_space_util -disk_space_used -disk_space_avail -disk_space_units Gigabytes
OK. I know! It’s not very simple so let’s take a look at third-party stuffs.
We chose New Relic (http://newrelic.com/) because they officially supports monitoring for EC2 instance in very simple way.
Installers for New Relic – Servers
All you need is to download the installer and install it on your VM. That’s it!
You will get the following dashboard after installing the New Relic installer on your server.
Plugins – New Relic
If you are not happy with default monitoring, you can look at thousands of plugins in “plugin central” or you can even create it on your own. (Note: We didn’t use “Amazon EC2” plugin until now but we are planning to test it in a few weeks time.)
Look cool and simple? Yes! it is.
New Relic has a few different plan that you can choose. As of now, we are using LIFE (a.k.a. Standard) version so we have only 24 hours for data retention. You can look at their prices in this link http://newrelic.com/pricing for details.
Last question. Is New Relic service expensive for server?
Here is what we found and I think it seems pretty okay.
Are you a EC2, Cloud Watch or New Relic user and got a tip to share? please feel free to drop a comment here. Thanks!