📘 CloudWatch → Slack Alerting Guide

Scope: Setup CloudWatch alarms for EC2 CPU utilization and ALB 4xx/5xx errors, centralize alarms via SNS + Lambda, and send enriched notifications to Slack with alarm name, resource details, and deep links.

1. Prerequisites

AWS Account with IAM admin or equivalent privileges.
Slack workspace with permission to create Apps or use Incoming Webhooks.
Python-based AWS Lambda runtime (3.9+ recommended).

2. Architecture Overview

CloudWatch Alarm → SNS Topic → Lambda → Slack Webhook

CloudWatch Alarms: Detect threshold breaches (CPU, ALB 4xx/5xx).
SNS Topic: Aggregates all alarms → single event pipeline.
Lambda Function: Parses alarm event, enriches with details, and posts to Slack.
Slack: Receives structured, actionable alerts.

3. Step 1 — Create an SNS Topic

aws sns create-topic --name cloudwatch-slack-topic

Note the Topic ARN (e.g., arn:aws:sns:ap-south-1:123456789012:cloudwatch-slack-topic).

4. Step 2 — Create CloudWatch Alarms

4.1 CPU Alarm (EC2)

Metric: AWS/EC2 → CPUUtilization.
Dimension: InstanceId=i-xxxxxxxx.
Threshold: > 80% for 5 minutes.

aws cloudwatch put-metric-alarm \
  --alarm-name "HighCPUUtilization" \
  --metric-name CPUUtilization \
  --namespace AWS/EC2 \
  --statistic Average \
  --period 300 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --dimensions Name=InstanceId,Value=i-0abcd1234efgh5678 \
  --alarm-actions arn:aws:sns:ap-south-1:123456789012:cloudwatch-slack-topic

4.2 ALB 5xx Alarm

Metric: AWS/ApplicationELB → HTTPCode_ELB_5XX_Count.
Dimension: LoadBalancer=app/my-alb/1234567890abcdef.
Threshold: > 10 errors in 5 minutes.

aws cloudwatch put-metric-alarm \
  --alarm-name "ALB-5XX-Errors" \
  --metric-name HTTPCode_ELB_5XX_Count \
  --namespace AWS/ApplicationELB \
  --statistic Sum \
  --period 300 \
  --threshold 10 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --dimensions Name=LoadBalancer,Value=app/my-alb/1234567890abcdef \
  --alarm-actions arn:aws:sns:ap-south-1:123456789012:cloudwatch-slack-topic

4.3 ALB 4xx Alarm

Same as 5xx, but HTTPCode_ELB_4XX_Count.

aws cloudwatch put-metric-alarm \
  --alarm-name "ALB-4XX-Errors" \
  --metric-name HTTPCode_ELB_4XX_Count \
  --namespace AWS/ApplicationELB \
  --statistic Sum \
  --period 300 \
  --threshold 100 \
  --comparison-operator GreaterThanThreshold \
  --evaluation-periods 1 \
  --dimensions Name=LoadBalancer,Value=app/my-alb/1234567890abcdef \
  --alarm-actions arn:aws:sns:ap-south-1:123456789012:cloudwatch-slack-topic

5. Step 3 — Setup Slack Webhook

Option A (Quick & Dirty)

Create an Incoming Webhook in Slack → get URL like: https://hooks.slack.com/services/TXXX/BXXX/XXXXXXXX

Option B (Recommended / Future-Proof)

Create a Slack App → Enable Incoming Webhooks → Install to Workspace → Copy Webhook URL.

6. Step 4 — Create Lambda Function

6.1 Lambda IAM Role

Grant Lambda access to:

AWSLambdaBasicExecutionRole.
AmazonSNSFullAccess (or SNS Subscribe).

6.2 Lambda Code (Python 3.9+)

import json
import urllib3
 
http = urllib3.PoolManager()
slack_webhook_url = "https://hooks.slack.com/services/XXXX/XXXX/XXXX"  # Replace
 
def lambda_handler(event, context):
    for record in event['Records']:
        message = json.loads(record['Sns']['Message'])
 
        alarm_name = message.get('AlarmName')
        description = message.get('AlarmDescription', 'No description')
        state = message.get('NewStateValue')
        reason = message.get('NewStateReason')
        region = message.get('Region')
        account = message.get('AWSAccountId')
 
        trigger = message.get('Trigger', {})
        metric = trigger.get('MetricName', 'N/A')
        namespace = trigger.get('Namespace', 'N/A')
        dimensions = {d['name']: d['value'] for d in trigger.get('Dimensions', [])}
        resource = ", ".join([f"{k}={v}" for k, v in dimensions.items()]) or "N/A"
 
        # Direct link to alarm in CloudWatch
        alarm_url = f"https://console.aws.amazon.com/cloudwatch/home?region={region}#alarmsV2:alarm/{alarm_name}"
 
        slack_message = {
            "attachments": [
                {
                    "color": "danger" if state == "ALARM" else "good",
                    "title": f"🚨 CloudWatch Alarm: {alarm_name}",
                    "title_link": alarm_url,
                    "fields": [
                        {"title": "State", "value": state, "short": True},
                        {"title": "Metric", "value": f"{namespace}/{metric}", "short": True},
                        {"title": "Resource", "value": resource, "short": True},
                        {"title": "Region", "value": region, "short": True},
                        {"title": "Account", "value": account, "short": True},
                        {"title": "Description", "value": description, "short": False},
                        {"title": "Reason", "value": reason, "short": False}
                    ],
                    "footer": "AWS CloudWatch",
                    "footer_icon": "https://a0.awsstatic.com/libra-css/images/logos/aws_logo_smile_1200x630.png"
                }
            ]
        }
 
        http.request(
            'POST',
            slack_webhook_url,
            body=json.dumps(slack_message),
            headers={'Content-Type': 'application/json'}
        )
 
    return "Message sent to Slack"

7. Step 5 — Connect SNS → Lambda

aws sns subscribe \
  --topic-arn arn:aws:sns:ap-south-1:123456789012:cloudwatch-slack-topic \
  --protocol lambda \
  --notification-endpoint arn:aws:lambda:ap-south-1:123456789012:function:CloudWatchToSlack

Then give Lambda permission to be invoked by SNS:

aws lambda add-permission \
  --function-name CloudWatchToSlack \
  --statement-id snsinvoke \
  --action "lambda:InvokeFunction" \
  --principal sns.amazonaws.com \
  --source-arn arn:aws:sns:ap-south-1:123456789012:cloudwatch-slack-topic

8. Step 6 — Testing

Manually set thresholds very low (e.g., CPU > 1%) to trigger alarms.
Verify messages in Slack include:
- Alarm Name
- State (ALARM / OK)
- Metric Name
- Resource ID
- AWS Region + Account
- Direct CloudWatch Console Link
- Reason

9. Enhancements (Optional)

Extend Lambda to include top failing ALB paths using ALB Access Logs + CloudWatch Logs Insights.
Route different alarms to different Slack channels by mapping AlarmName → Slack Webhook.
Add severity levels (critical, warning, info) based on alarm type.

✅ Executive Summary

CPU alarms tell you which EC2 instance is overutilized.
ALB 4xx/5xx alarms tell you which load balancer is serving bad requests.
Slack notifications are enriched with alarm name, metric, resource ID, and direct AWS links.
A single Lambda function manages all alarms, making operations lean and scalable.

api.slack.com hooks.slack.com

Slack API Documentation for Webhooks APP

https://docs.slack.dev/quickstart (opens in a new tab)

Eks Grafana Athena S3 Alb Logs Athena