Amazon CloudWatch Metric Streams in near real time

New Relic integrates with Amazon CloudWatch Metric Streams to provide a solution where deep application and infrastructure monitoring rely on the AWS Cloud.

This blog post first appeared on the New Relic blog, on November 1st, 2022.

The AWS Well-Architected Framework helps you understand the benefits and risks of decisions you make while building workloads on AWS. You learn best practices for designing and operating reliable, secure, efficient, and cost-effective workloads in the cloud. It provides a way to consistently measure your operations and architectures against best practices and identify areas for improvement. Well-Architected workloads designed with operations in mind dramatically increase the likelihood of business success.

To prepare for operational excellence, you must understand your workloads and their expected behaviors. Then you can build procedures to support your workloads and get better insight into their status. The first step for achieving operational excellence is to design telemetry monitoring for your workloads to provide the information necessary to understand their internal state (for example, metrics, logs, events, and traces). You build an overall solution for full observability across all components of your application architecture. In AWS, you can emit and collect logs, metrics, and events from your applications and workloads components, helping you understand their internal state and health with Amazon CloudWatch. Then you use this data to understand how your application and underlying components interact to analyze issues and performance.

Amazon CloudWatch Metric Streams

New Relic provides a platform to collect, explore, and alert on all your telemetry data from AWS in one place. The platform also provides API endpoints to ingest the telemetry data conveniently. This solution diagram shows how CloudWatch Metric Streams and New Relic work together:

When working with telemetry data, it’s crucial to use up-to-date data. You need to immediately visualize and understand the state of your system, especially during operational events. Previously, you needed to ensure you continuously queried CloudWatch to keep the in-house systems refreshed with the latest data. With CloudWatch Metric Streams, you don’t have to carry the burden of refreshing data. 

Steven Osborne, Director Of Engineering at Workiva says, “Every minute counts with operational incidents, and metric streaming has unlocked valuable minutes to detect, respond to, and mitigate service level-concerns even faster than before!”

This blog post walks through step-by-step instructions to detect and quickly respond to operational events using Amazon CloudWatch Metric Streams and New Relic. 

You can walk through each step, or use this AWS CloudFormation template, which automates step 1, step 2, and step 4.

Step 1: Create an Amazon Kinesis data firehose delivery stream

Open the CloudWatch console, and select the region you want. Then, go to Kinesis and select Create a new data firehose delivery stream. Follow the prompts in Kinesis. This example creates the delivery stream in us-east-1. You can use these values:

• Delivery stream name: Enter new-relic-metric-stream-us-east-1. (Use any name according to your organization’s naming conventions.)
• Source: Select Direct PUT or other sources.
• Destination: Select Third-party service provider and New Relic.
• HTTP endpoint URL: Select New Relic Metrics–US. If your AWS region is in the European Union, select New Relic Metrics–EU.
• API key: Enter your New Relic License key.
• Create or update the AWS Identity and Access Management (IAM)  role.
• Buffer size: Enter 1MB.

Step 2: Create IAM Role and establish trust for New Relic integration

Use the AWS IAM console to create a role. This new role allows New Relic to retrieve monitoring data from your AWS account securely. Open the IAM console and click Create Role. If you do not have permission to access this part of your AWS account, contact your AWS account administrator.

• Select the following role type: Another AWS account.
• Enter 754728514883 for the Account ID.
• Select the box to Require an external ID.
• Enter 3093930 for the External ID and click Next: Permissions.
• Enter ReadOnlyAccess in the search box.
• Select the box for the ReadOnlyAccess policy and select Next: Review.
• Enter NewRelicInfrastructure-Integrations for the role name.
• Select Create Role and Copy the Role ARN.
• Optional: Enabling the AWS Config service allows New Relic to enrich metrics with metadata information from the different services:
  • Add the inline policy in the Permissions tab.
  • Create a policy for the Config service, and provide the BatchGetResourceConfig read permission. You can use the following permission statement in JSON:

{
  "Statement": [
    {
      "Action": [
        "config:BatchGetResourceConfig"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ],
  "Version": "2012-10-17"
}

Then enter a Name for the policy (for example NewRelicConfig) and select Create policy.

Step 3: Enable integration on New Relic

Now you create the New Relic side of the integration using Add more data on the New Relic platform home page. 

Tip: You can also add an AWS account from the Infrastructure > AWS page. Make sure to select the Metric Streams integration mode and follow the steps.

Step 4: Stream Amazon CloudWatch metric streams data

Open the CloudWatch console, and select the region you want. Then go to CloudWatch and select Create a metric stream. Select the following data:

• Select namespaces you want to stream: All metrics.
• Select a configuration option:
  • Select an existing Firehose owned by your account, then select the one created in Step 1.
  • Use the custom metric stream name: new-relic-metric-stream.

Within a few minutes, the CloudWatch metrics data starts flowing in your New Relic account.

Step 5: Set up policies to alert

Open the New Relic platform and select the Alerts & AI tab. Then take these actions:

• Use an existing policy or create a new alert policy.
• Add a condition to your policy: Select NRQL for the category.
• Give the condition a name.
• Define the signal using this NRQL query:

SELECT max(aws.ec2.CPUUtilization) FROM Metric FACET tags.Name

• Set the condition thresholds as you like.
• Save the condition.

Step 6: Testing

To ensure everything is set up correctly, check the Amazon Metric Streams status page from the one.newrelic.com > Infrastructure > AWS page. Check the configuration using the Account Status dashboard and explore the Metric Streams data. The Account Status dashboard provides a view across metrics, namespaces, metric names, and total metrics by namespace.

Next, launch an Amazon EC2 instance and run a command that causes the CPU utilization to spike up. The intention here is to test and confirm if you can get alerts in near real-time using CloudWatch Metric Streams and New Relic. You can use the t2.nano instance type. When created, SSH in the instance and run dd if=/dev/zero of=/dev/null, which causes the CPU to spike. Within minutes, you should see the notification from New Relic.