AWS OTEL is a secure, production-ready, AWS-supported distribution of the OpenTelemetry project.
With AWS Distro for OpenTelemetry, you can instrument your applications just once to send correlated metrics and traces to multiple AWS and Partner monitoring solutions.
It also collects metadata from your AWS resources and managed services, so you can correlate application performance data with underlying infrastructure data, reducing the mean time to problem resolution.
And this repository contains a practical example about how to use AWS OTEL for tracing when we have a series of distributed .NET apps.
The repository contains the following applications:
-
App1.WebApi is a .NET6 Web API with 2 endpoints.
- The
/http
endpoint makes an HTTP request to the App2/dummy
endpoint. - The
/publish-message
endpoint queues a message into an AWS ActiveMQ Rabbit queue.
- The
-
App2.RabbitConsumer.Console is a .NET6 console app.
- Dequeues messages from the Rabbit queue and makes a HTTP request to the App3
/s3-to-event
endpoint with the content of the message.
- Dequeues messages from the Rabbit queue and makes a HTTP request to the App3
-
App3.WebApi is a .NET6 Web API with 2 endpoints.
- The
/dummy
endpoint returns a fixed "Ok" response. - The
/s3-to-event
endpoint receives a message via HTTP POST, stores it in an S3 bucket and afterwards publishes the message as an event into an AWS SQS queue.
- The
-
App4.SqsConsumer.HostedService is a .NET6 Worker Service.
- A Hosted Service reads the messages from the AWS SQS queue and stores it into a DynamoDb table.
This repository has a CDK app that will create all these resources.
This demo uses the following AWS resources:
- A VPC (CIDR Address: 10.30.0.0/16) with a public subnet.
- An internet facing AmazonMQ RabbitMQ cluster.
- An S3 bucket.
- An SQS Queue.
- A Dynamo table.
- An IAM User with the following managed policies:
AmazonS3FullAccess
AmazonSQSFullAccess
AmazonDynamoDBFullAccess
- Also uses an inline policy with the following permissions:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:PutLogEvents",
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogStreams",
"logs:DescribeLogGroups",
"xray:*",
"ssm:GetParameters"
],
"Resource": "*"
}
]
}
This policy is needed by the AWS OTEL Collector.
The apps are using the following package versions:
<PackageReference Include="OpenTelemetry" Version="1.2.0-rc1" />
<PackageReference Include="OpenTelemetry.Extensions.Hosting" Version="1.0.0-rc8" />
<PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc8" />
<PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc8" />
<PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.2.0-rc1" />
<PackageReference Include="OpenTelemetry.Contrib.Extensions.AWSXRay" Version="1.1.0" />
<PackageReference Include="OpenTelemetry.Contrib.Instrumentation.AWS" Version="1.0.1" />
The repository contains a CDK app that creates the AWS Resources needed to run the demo and it also contains a docker-compose file that starts up the 4 apps and the AWS OTEL collector.
There is some work that needs to be done in the docker-compose file before you execute it:
- If you take a look at the compose file you'll see that there are a few values that MUST be replaced:
<ADD-AWS-USER-ACCESS-KEY>
<ADD-AWS-USER-SECRET-KEY>
<ADD-AMAZONMQ-RABBIT-HOST-ENDPOINT>
<ADD-S3-BUCKET-NAME>
<ADD-SQS-URI>
You can find the correct values in the output of the CDK app. Here's an example of how the output of the AWS CDK app looks like:
And here's an example of how the docker-compose looks like after replacing the placeholder values:
version: '3.4'
networks:
tracing:
name: tracing-network
services:
otel:
image: amazon/aws-otel-collector:latest
command: --config /config/config.yml
volumes:
- ./aws-otel-collector-config:/config
environment:
- AWS_ACCESS_KEY_ID=AKIA5S6L5S6L5S6L5S6L
- AWS_SECRET_ACCESS_KEY=GO7BvT9IBLb4NudL0aGO7BvT9IBLb4NudL0a
- AWS_REGION=eu-west-1
ports:
- 4317:4317
networks:
- tracing
app1:
build:
context: ./App1.WebApi
ports:
- "5000:80"
networks:
- tracing
depends_on:
- otel
- app3
environment:
RABBITMQ__HOST: b-27c79732-ce9d-47dc-8c51-13eec94c267e.mq.eu-west-1.amazonaws.com
RABBITMQ__USERNAME: specialguest
RABBITMQ__PASSWORD: P@ssw0rd111!
APP3ENDPOINT: http://app3/dummy
OTLP__ENDPOINT: http://otel:4317
OTEL_RESOURCE_ATTRIBUTES: service.name=App1
app2:
stdin_open: true
tty: true
build:
context: ./App2.RabbitConsumer.Console
networks:
- tracing
depends_on:
- otel
- app3
environment:
RABBITMQ__HOST: b-27c79732-ce9d-47dc-8c51-13eec94c267e.mq.eu-west-1.amazonaws.com
RABBITMQ__USERNAME: specialguest
RABBITMQ__PASSWORD: P@ssw0rd111!
APP3URIENDPOINT: http://app3
OTLP__ENDPOINT: http://otel:4317
OTEL_RESOURCE_ATTRIBUTES: service.name=App2
app3:
build:
context: ./App3.WebApi
ports:
- "5001:80"
networks:
- tracing
depends_on:
- otel
environment:
OTLP__ENDPOINT: http://otel:4317
OTEL_RESOURCE_ATTRIBUTES: service.name=App3
S3_BUCKET_NAME: aws-otel-demo-s3-bucket-17988
SQS__URI: https://sqs.eu-west-1.amazonaws.com/7777777/aws-otel-demo-sqs-queue
AWS_ACCESS_KEY_ID: AKIA5S6L5S6L5S6L5S6L
AWS_SECRET_ACCESS_KEY: GO7BvT9IBLb4NudL0aGO7BvT9IBLb4NudL0a
app4:
build:
context: ./App4.SqsConsumer.HostedService
networks:
- tracing
depends_on:
- otel
environment:
OTLP__ENDPOINT: http://otel:4317
OTEL_RESOURCE_ATTRIBUTES: service.name=App4
SQS__URI: https://sqs.eu-west-1.amazonaws.com/7777777/aws-otel-demo-sqs-queue
AWS_ACCESS_KEY_ID: AKIA5S6L5S6L5S6L5S6L
AWS_SECRET_ACCESS_KEY: GO7BvT9IBLb4NudL0aGO7BvT9IBLb4NudL0a
To summarize, if you want to run this demo you'll need to do the following steps:
- Execute the CDK app on your AWS account.
- Replace the values on the docker-compose.
- Execute
docker-compose up
.
Here's the XRay trace output:
And the XRay Service Map output: