I’m frustrated. A major service of AWS is broken for 65 days. The Simple Notification Service (SNS) delivers messages to HTTPS subscriptions with a delay of more than 30 minutes. That issue impacts our SaaS business. But AWS did not fix the problem yet.
Aws Sns To Slack
You will find the latest updates on the issue at the end of the blog post.
The Problem
In CloudWatch, you define alarms to send a message to an SNS topic if the monitoring data gets out of normal bounds. Finally, you connect a Lambda function to the SNS topic to trigger a function execution. The Lambda function calls the Slack API to send a message. The following figure shows the data flow. Configure an AWS Chatbot client for a slack channel You can create notification rules that use an AWS Chatbot client as a target. If you create a client for a Slack channel, you can use this client directly as a target in the workflow for creating a notification rule.
Aws Cloudwatch Sns Slack
Our SaaS business runs on a Serverless architecture. Our customers send all kinds of alarms from their AWS infrastructure to our chatbot. To do so, our solution configures CloudWatch alarms and an SNS topic within our customer’s AWS accounts. The SNS topic forwards all incoming alarms to our API Gateway by using an HTTP subscription.
On September 1st, a customer wrote in: they observed that CloudWatch alarms showed up in Slack with a delay of more than 30 minutes. A monitoring and incident management solution is kind of worthless with delayed alarms. Therefore, I started investigating immediately.
First of all, I had a look at the AWS Service Health Dashboard. All systems operating normally.
Next, I analyzed our log messages to track down the issue. But I could not find any delayed messages—incoming alarms were delivered to Slack within milliseconds after arriving at our API Gateway.
I was wondering how to find out whether SNS caused the delay. But how to investigate an issue like that? Luckily, I stumbled upon delivery status logging. A SNS topic is capable of writing delivery logs to CloudWatch Logs. The perfect way to debug a problem like that.
I found log messages similar to this one. SNS sent a message to api.marbot.io, and our API Gateway answered with status code 204. SNS tried to deliver the message once. The important information is dwellTimeMs = 2748244. It took SNS about 45 minutes to send the alarm to our backend.
As we all know, things go wrong. That’s why monitoring and alerting are essential topics. Wouldn’t it be nice, if problems in your AWS account would show up in Slack? So you can react quickly while using your favorite messaging tool. In this blog post, you will learn how you can turn CloudWatch Alarms into Slack messages like this:
How it works
On AWS, everything sends monitoring data (CPU utilization, estimated monthly charges, …) to CloudWatch. In CloudWatch, you define alarms to send a message to an SNS topic if the monitoring data gets out of normal bounds. Finally, you connect a Lambda function to the SNS topic to trigger a function execution. The Lambda function calls the Slack API to send a message. The following figure shows the data flow:
To deploy the components in the figure, you will use the Serverless Application Model (SAM). If you are not interested in implementing this on your own, give our Slack chatbot a try. Never miss an alert from your AWS infrastructure with marbot!
Implementing the Lambda function
You will use Node.js to implement the Lambda function. To send a request to the Slack API, you have to make an HTTPS request. The request module is easy to use, but I wanted a variant of the module that returns promises to avoid callback hell. That’s why I used request-promise-native. The Slack webhook URL is passed in as an environment variable that you define later in the CloudFormation template.
Messages delivered from SNS to the Lambda function will look like this:
You need to convert the format into the Slack message format.
Finally, each Lambda function needs a handler function. The handler function takes 3 parameters: