How to monitor Glasnostic appliances using CloudWatch
As is the case with any critical component in a production environment, we recommend that you run two Glasnostic instances and continuously monitor their health to ensure Glasnostic is always available. In this article, we’ll show you how to do this with AWS Cloudwatch.
First off, we’ll set up Cloudwatch to alert us when an instance has become unresponsive.
Then click on “GatewayELB” followed by “Per LB, per TG Metrics” to show the metrics available for each GWLB.
Select “HealthyHostCount,” then click the “Graphed metrics” tab. This shows the number of Glasnostic instances the load balancer sends traffic to over time:
Although the Glasnostic appliance consists of two Glasnostic instances, the only high-availability mode supported by GWLB at this time is a “Hot-Cold” setup. The number of healthy Glasnostic instances should therefore always be “1” and we want to create an alert if it drops below “1.” To do so, click the bell icon under “Actions”, select “Static” and “Lower,” and enter “1” in the box:
Click “Next” to move to the “Notification” dialog, where you can specify how you want to receive notifications. Here we chose an SNS Notification with a topic name of “GlasnosticHealth” and specify an email address that should receive this notification:
After clicking “Create topic” and confirming the notification by clicking “Next,” name the alert and click “Next” again.
Finally, review the new settings and create the alert.
You’ll now receive a notification to the email address provided that confirms the alert subscription.
Once the alert is set up, let’s test if our monitoring works.
We’ll test whether our monitoring works by stopping the two Glasnostic instances and then booting them up again.
Warning: This will lead to a short downtime of about 10 minutes, so don’t do this with a production system!
Open the AWS EC2 console and select “Instances” in the navigation pane.
Search for “Glasnostic,” then select both instances:
Open the “Instance state” drop-down menu and select “Stop instance.”
Confirm by clicking “Stop” on the following dialog:
Once the instances have shut down, we should receive an alert email at the address configured above.
Restart the two instances by selecting them both and clicking “Start instance” in the “Instance state” drop-down. The instances will continue to process traffic automatically after reboot.
Note that alerts will be triggered only if both instances have failed. If merely one instance has failed, the appliance will fail over to the secondary instance automatically without triggering an alert.