Amazon EC2 + CloudWatch + New Relic: Monitoring, Alert Notifications

We have been on Ec2 for almost two years but recently, we needed to monitor a few stuffs on server so we did a bit of experiences on that and deployed it a couple of months back. We found that we had a pretty good result so I am going to share it here as usual.

Purpose

To monitor the following things on Amazon Ec2 virtual machine and send the alerts based on some conditions.

  • CPU Utilization
  • Memory Available
  • Disk Space Available
  • Network
  • VM Status

Methods of monitoring

There are a few ways to monitor the instances of EC2 but we chose to use Amazon CloudWatch and New Relic.

  • Cloud Watch
  • New Relic

Amazon Cloud Watch

The reason is simple. We chose it because Amazon provides it. There is no cost for monitoring some basic stuffs (Amazon CloudWatch Pricing) so it fits us well.

Basic Monitoring metrics (at five-minute frequency) for Amazon EC2 instances are free of charge, as are all metrics for Amazon EBS volumes, Elastic Load Balancers, and Amazon RDS DB instances.
New and existing customers also receive 10 metrics (applicable to Detailed Monitoring for Amazon EC2 instances or Custom Metrics), 10 alarms, and 1 million API requests each month at no additional charge.

Amazon EC2 Console – Default Monitoring

This is what you will see when you open up the EC2 console. You can enable the monitoring and create the alarms in this console as well.


Amazon EC2 - Monitoring Page

CloudWatch – Monitoring

Once you have enabled the default monitoring in EC2 console, you will see the following metrics in CloudWatch console.

CloudWatch Console for EC2

 

Of course, this is not what we need. We need to monitor the disk space, memory utilization and etc. So we need to create some custom scripts for that.

AWS SDK for .NET

“AWS SDK for .NET” allows you to create the scripts that can create the custom metric on CloudWatch. You can download the SDK from this link ( http://aws.amazon.com/sdkfornet/). We want to monitor the disk space and memory which doesn’t include in EC2 default monitoring so we need to install this SDK and create the script for that.

Amazon CloudWatch Monitoring Scripts for Microsoft Windows Server

You can roll your own script for monitoring what you want to monitor but it’s always a good idea to google before creating your own because if your requirements is very common then someone might be already created the script for that.

I found a brunch of monitoring scripts from this link ( http://aws.amazon.com/code/7932034889155460 ). This package includes the scripts for the following metrics.

  • Memory Utilization (%)
  • Memory Used (MB)
  • Memory Available (MB)
  • Page File Utilization (%)
  • Page File used (MB)
  • Page File available (MB)
  • Disk Space Utilization (%)
  • Disk Space Used (GB)
  • Disk Space Available (GB)
  • Perfmon Counters.

Amazon Access Key ID and Secret Access Key

To run those scripts, you will probably need to provide the access key and it wasn’t that straight forward to find your own secret access key on Amazon so I captured the screenshots of navigation.

3. Security Credentials 4. Amazon Access Kerys ID and Secret Access Key

 

Bugs in Amazon CloudWatch Monitoring Scripts

You will get the following error when you try to load the Amazon powershell module.

Error Screenshot

5. Default Powersshell Error

Error Message in text (for Google. of course! )

Windows PowerShell
Copyright (C) 2012 Microsoft Corporation. All rights reserved.

Import-Module : The specified module ‘C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell.psd1’ was not loaded
because no valid module file was found in any module directory.
At C:\Users\michael.sync\Documents\WindowsPowerShell\Microsoft.PowerShell_profile.ps1:1 char:1
+ Import-Module “C:\Program Files (x86)\AWS Tools\PowerShell\AWSPowerShell.psd1”
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ResourceUnavailable: (C:\Program File…PowerShell.psd1:String) [Import-Module], FileNot
FoundException
+ FullyQualifiedErrorId : Modules_ModuleNotFound,Microsoft.PowerShell.Commands.ImportModuleCommand

PS C:\Users\michael.sync>

Solution

This issue occurs because of the wrong path in powershell profile. Look the screenshot below for the file name and its location and you can fix the path.

6. PowerShell Error RootCause

Note: $env:psmodulePath is the automatic variable which holds the path used to discover modules. If it’s not set, PowerShell looks in c:\windows\system32\WindowsPowerShell\1.0\modules and MyDocuments\WindowsPowerShell\modules

After fixing the wrong, you should be able to run any script from |Amazon monitoring scripts”.

7. PowerShell

if you can manage to run the scripts that you requires the metrics to be appeared then you can trigger those scripts from windows scheduler every 5 mins or so.

And then you will see new custom metrics as below in CloudWatch dashboard. You can go ahead and create some alerts on new custom metric.

8. Windows Custom Matrix

Here is the scripts that I am using for monitoring the memory utilization and disk space.


.\mon-put-metrics-mem.ps1 -aws_credential_file C:\Users\michael.sync\Downloads\AmazonCloudWatchMonitoringWindows\awscreds.conf -mem_util -mem_used -mem_avail -page_avail -page_used -page_util -memory_units Megabytes

.\mon-put-metrics-disk.ps1 -aws_credential_file C:\Users\michael.sync\Downloads\AmazonCloudWatchMonitoringWindows\awscreds.conf -disk_drive C:, D: -disk_space_util -disk_space_used -disk_space_avail -disk_space_units Gigabytes

OK. I know! It’s not very simple so let’s take a look at third-party stuffs.

New Relic

newrelic_logo-300x74

We chose New Relic (http://newrelic.com/) because they officially supports monitoring for EC2 instance in very simple way.

Installers for New Relic – Servers

All you need is to download the installer and install it on your VM. That’s it!

New Reclic EC2

 

You will get the following dashboard after installing the New Relic installer on your server.

 

New Reclic Chart for EC2

Plugins – New Relic

If you are not happy with default monitoring, you can look at thousands of plugins in “plugin central” or you can even create it on your own. (Note: We didn’t use “Amazon EC2” plugin until now but we are planning to test it in a few weeks time.)

New Reclic PLugin

Look cool and simple? Yes! it is.

New Relic has a few different plan that you can choose. As of now, we are using LIFE (a.k.a. Standard) version so we have only 24 hours for data retention. You can look at their prices in this link http://newrelic.com/pricing for details.

New Reclic Price

 

Last question. Is New Relic service expensive for server?

Here is what we found and I think it seems pretty okay.

Is New Reclic expensive

Are you a EC2, Cloud Watch or New Relic user and got a tip to share? please feel free to drop a comment here. Thanks!

WebDeploy 3 – Error in publishing website to Amazon EC2

Here is the error that I got when I publish website from Azure VM to Amazon EC2 VM. I am using the following MSBuild command to do the deployment and I got Web Deploy 3.0 installed on EC2 VM.

/p:Configuration=Release /p:OutputPath=bin /p:VisualStudioVersion=11.0 /p:DeployOnBuild=True /p:DeployTarget=MSDeployPublish /p:MsDeployServiceUrl=https://my-super-cool-server/msdeploy.axd /p:AllowUntrustedCertificate=True /p:DeployIisAppPath=YourWebSiteInstanceName /p:MSDeployPublishMethod=WMSVC /p:username=root /p:password=SuperComplexRootPassword

Error Message [VSMSDeploy] C:\Program Files (x86)\MSBuild\Microsoft\VisualStudio\v11.0\Web\Microsoft.Web.Publishing.targets(4196, 5): error ERROR_DESTINATION_NOT_REACHABLE: Web deployment task failed. (Could not connect to the remote computer (my-super-cool-server””). On the remote computer, make sure that Web Deploy is installed and that the required process (“Web Management Service”) is started. Learn more at: http://go.microsoft.com/fwlink/?LinkId=221672#ERROR_DESTINATION_NOT_REACHABLE.) [14:58:46][MyWebsite\MyWebsite.csproj.teamcity] Project MyWebsite\MyWebsite.csproj.teamcity failed. VSMSDeploy-update Solution As this post is not about how to use WebDeploy to deploy website, I am not going to write the detailed steps here. If you are new to using webdeploy, please read this post “Automating Deployment with Microsoft Web Deploy” The reason that I got this error is that the default for Web Management Service is not enabled by default in EC2. The solution is to enable the port 80 (which you might already enabled it for your website) for Web Deployment Agent and port 8172 for Web Management Service in both the server’s firewall and EC2 Security Group.

  • Web Deployment Agent Service : Port 80
  • Web Management Service : Port 8172

EC2 Security Group Hope it helps for those who are using MS deploy for publishing website on IIS. Reference: