aws-observability / aws-otel-python Goto Github PK
View Code? Open in Web Editor NEWAWS Distro for OpenTelemetry Python
Home Page: https://aws-otel.github.io/
License: Apache License 2.0
AWS Distro for OpenTelemetry Python
Home Page: https://aws-otel.github.io/
License: Apache License 2.0
During Soak Tests execution, a performance degradation was revealed for commit 8a92ecf of the refs/heads/main
branch for the (flask, auto) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
Hello, I opened an issue regarding this on the opentelemetry-python-contrib repo but thought it would also be relevant here since it would be helpful to add instructions to this documentation.
I am trying to use the new OTEL_EXPERIMENTAL_RESOURCE_DETECTORS
environment variable (introduced in open-telemetry/opentelemetry-python#3181) with the AWS resource detectors in opentelemetry-sdk-extension-aws
but it is not detecting the resource detectors.
Please see this issue for more information: open-telemetry/opentelemetry-python-contrib#2370
Update the Python sample app to have feature parity with the Java sample app. This is part of #63
During Soak Tests execution, a performance degradation was revealed for commit 653f117 of the refs/heads/main
branch for the (flask, auto) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
After the Soak Tests completed, a performance degradation was revealed for commit 01c4956 of the refs/heads/main
branch for the (flask, manual) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
The threshold violation should also be noticeable on our graph of Soak Test average results per commit.
During Soak Tests execution, a performance degradation was revealed for commit e8388c3 of the refs/heads/main
branch for the (flask, manual) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
After the Soak Tests completed, a performance degradation was revealed for commit 01c4956 of the refs/heads/main
branch for the (flask, auto) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
The threshold violation should also be noticeable on our graph of Soak Test average results per commit.
During Soak Tests execution, a performance degradation was revealed for commit b219983 of the refs/heads/main
branch for the (flask, none) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
During Soak Tests execution, a performance degradation was revealed for commit 653f117 of the refs/heads/main
branch for the (flask, manual) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
When Python SDK goes GA for metrics make sure that:
Note: this SDK is a dependency of our ADOT Lambda offering.
After the Soak Tests completed, a performance degradation was revealed for commit 8680fa3 of the refs/heads/main
branch for the (flask, none) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
The threshold violation should also be noticeable on our graph of Soak Test average results per commit.
Currently, the sample app in this repo which tests automatic instrumentation with Flask (i.e. sample-apps/auto-instrumentation/flask
) cannot provide Metrics because automatic Metric tracing is not yet supported in OpenTelmetry Python. See the issue here.
This means we have a 2 code changes to do in the app once that issue is resolved:
aws-otel-python/sample-apps/utils/setup_metrics.py
Lines 16 to 18 in 4f81ff0
aws-otel-python/sample-apps/utils/setup_metrics.py
Lines 48 to 55 in 4f81ff0
After the Soak Tests completed, a performance degradation was revealed for commit 01c4956 of the refs/heads/main
branch for the (flask, none) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
The threshold violation should also be noticeable on our graph of Soak Test average results per commit.
Description
In case of architecture:
Flow:
Both functions are auto instrumented by AWS ADOT Instrumentation deployed as Lambda layer
Expected behaviour:
Only 1 trace with:
Current behaviour:
There are two separated traces:
Function A
with a CLIENT span (sending message to SQS queue)Function B
but only with span related to RDS callNote:
What I found is lack of AWSTraceHeader
in SQS attributes for AWS OTEL Python Instrumentation. I believe this attribute is responsible for context propagation.
When AWS X Ray is used as tracing then AWSTraceHeader
is set.
During Soak Tests execution, a performance degradation was revealed for commit e8388c3 of the refs/heads/main
branch for the (flask, none) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
The AWS X-Ray developer guide talks about Trace ID Injection. Trace ID Injection allows services like CloudWatch Service to link X-Ray segments to X-Ray segment Service Graphs. This is accomplished by injecting the current AWS X-Ray segment Trace ID into the log that is sent to CloudWatch under the AWS-XRAY-TRACE-ID
key (the key is not important since the query is done on the Trace ID, but this is just an example). For example, as seen in the docs:
2019-09-10 18:58:30.844 [nio-5000-exec-4] AWS-XRAY-TRACE-ID: 1-5d77f256-19f12e4eaa02e3f76c78f46a@1ce7df03252d99e1 WARN 1 - Your logging message here
The problem is, that we do not currently have a package that does this. This is unlike ADOT Java, which publishes a package for log4j and other logging libraries as well.
We should distribute a package like this which gives an OpenTelemetry Instrumentor
that injects X-Ray formatted Trace IDs from the current OpenTelemetry Python span. Since we want to do X-Ray specific Trace IDs, the package should be hosted in this repo and not OTel Python Contrib.
NOTE: We need to set up publishing to PIP before we can work on this issue.
After the Soak Tests completed, a performance degradation was revealed for commit 8a92ecf of the refs/heads/main
branch for the (flask, manual) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
The threshold violation should also be noticeable on our graph of Soak Test average results per commit.
During Soak Tests execution, a performance degradation was revealed for commit 404aa6a of the refs/heads/main
branch for the (flask, none) Sample App. Check out the Action Logs from the Soak Testing
workflow run on GitHub to view the threshold violation.
Snapshots of the Soak Test run are available on the gh-pages branch. These are the snapshots for the violating commit:
For OTel Python right now, if someone wants to use OTel SDK to send traces to AWS X-Ray, there are many packages that they have to install & environment variables they have to set. Even with auto instrumentation from OpenTelemetry Python, there are many packages to install to get started tracing with AWS X-Ray:
i.e.
boto3
requests==2.23.0
Flask==1.1.2
opentelemetry-api==0.16b1
opentelemetry-sdk==0.16b1
opentelemetry-instrumentation==0.16b1
opentelemetry-exporter-otlp==0.16b1
opentelemetry-sdk-extension-aws==0.16b1
We should consider writing a wrapper that will install all these packages for the customer and allow them to simply run the wrapper with config parameters to their needed settings. The wrapper will in turn run their application, which will make their application instrumented with OTel for AWS X-Ray.
Some questions:
In other repos:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.