How to monitor Glasnostic appliances using CloudWatch

All figures: code up like normal figures, but without caption. Filenames can be generic but alt texts should be descriptive (e.g. “Screenshot of AWS Console showing …”)

As is the case with any critical component in a production environment, we recommend that you run two Glasnostic instances and continuously monitor their health to ensure Glasnostic is always available. In this article, we’ll show you how to do this with AWS Cloudwatch.

Setup

First off, we’ll set up Cloudwatch to alert us when an instance has become unresponsive.

  1. In the CloudWatch Management Console, click on “Metrics” on the left.

  2. Then click on “GatewayELB” followed by “Per LB, per TG Metrics” to show the metrics available for each GWLB.

  3. Select “HealthyHostCount,” then click the “Graphed metrics” tab. This shows the number of Glasnostic instances the load balancer sends traffic to over time:

    Screenshot of AWS console showing Metrics in CloudWatch
  4. Although the Glasnostic appliance consists of two Glasnostic instances, the only high-availability mode supported by GWLB at this time is a “Hot-Cold” setup. The number of healthy Glasnostic instances should therefore always be “1” and we want to create an alert if it drops below “1.” To do so, click the bell icon under “Actions”, select “Static” and “Lower,” and enter “1” in the box:

    Screenshot of AWS console showing how to configure Conditions of alert
  5. Click “Next” to move to the “Notification” dialog, where you can specify how you want to receive notifications. Here we chose an SNS Notification with a topic name of “GlasnosticHealth” and specify an email address that should receive this notification:

    Screenshot of AWS console showing how to configure Notification
  6. After clicking “Create topic” and confirming the notification by clicking “Next,” name the alert and click “Next” again.

  7. Finally, review the new settings and create the alert.

You’ll now receive a notification to the email address provided that confirms the alert subscription.

Once the alert is set up, let’s test if our monitoring works.

Test

We’ll test whether our monitoring works by stopping the two Glasnostic instances and then booting them up again.

Warning: This will lead to a short downtime of about 10 minutes, so don’t do this with a production system!

  1. Open the AWS EC2 console and select “Instances” in the navigation pane.

  2. Search for “Glasnostic,” then select both instances:

    Screenshot of AWS console showing list of Glasnostic instances
  3. Open the “Instance state” drop-down menu and select “Stop instance.”

  4. Confirm by clicking “Stop” on the following dialog:

    Screenshot of AWS console showing dialog stopping instance
  5. Once the instances have shut down, we should receive an alert email at the address configured above.

  6. Restart the two instances by selecting them both and clicking “Start instance” in the “Instance state” drop-down. The instances will continue to process traffic automatically after reboot.

Note that alerts will be triggered only if both instances have failed. If merely one instance has failed, the appliance will fail over to the secondary instance automatically without triggering an alert.

Manage the complex behaviors of modern microservice architectures