Giter Club home page Giter Club logo

aws-xray-java-agent's Introduction

Build Status

Screenshot of the AWS X-Ray console

AWS X-Ray Java Agent

The AWS X-Ray Java Agent is a drop-in solution that enables the propagation of X-Ray traces within your Java web applications and servlet-based microservices. This includes automatic tracing for supported frameworks and libraries, including Spring, the AWS SDK, Apache HTTP clients, and JDBC-based SQL queries. The agent enables you to use the X-Ray SDK out of box, and requires no code changes to enable the basic propagation of traces. See the chart below for the current feature parity between the AWS X-Ray SDK and the AWS X-Ray Java Agent.

The X-Ray Java Agent is implemented using the DiSCo library, a toolkit for building Java Agents in distributed environments.

Compatibility Chart

Feature X-Ray SDK X-Ray Agent
AWS SDK V1 Instrumentation (Confirmed on 1.11.x)
AWS SDK V2 Instrumentation
Centralized Sampling
Automatic Multi-threaded Support
Custom manual instrumentation
SQL Queries
Plugins
Apache HTTP Client
HttpServlet
Lambda Layers
Log Injection
Spring Framework

Prerequisites

The AWS X-Ray Java Agent is compatible with Java 8 and 11. You must be able to modify the JVM arguments of your application to use the agent.

Installation

You can download the latest version of the X-Ray Agent here, or you can browse the releases to download earlier versions.

Alternatively, you can download the agent from Maven Central by adding a dependency on it. This approach is not recommended because you will be responsible for assembling all the JARs into the required file structure. To depend on the agent from your project, just add these dependencies:

<dependencies>
    <dependency>
        <groupId>software.amazon.disco</groupId>
        <artifactId>disco-toolkit-bom</artifactId>
        <version>0.11.0</version>
        <type>pom</type>
        <scope>import</scope>
    </dependency>
    <dependency>
        <groupId>software.amazon.disco</groupId>
        <artifactId>disco-java-agent</artifactId>
    </dependency>
    <dependency>
        <groupId>software.amazon.disco</groupId>
        <artifactId>disco-java-agent-aws-plugin</artifactId>
    </dependency>
    <dependency>
        <groupId>software.amazon.disco</groupId>
        <artifactId>disco-java-agent-sql-plugin</artifactId>
    </dependency>
    <dependency>
        <groupId>software.amazon.disco</groupId>
        <artifactId>disco-java-agent-web-plugin</artifactId>
    </dependency>
    <dependency>
        <groupId>com.amazonaws</groupId>
        <artifactId>aws-xray-agent-plugin</artifactId>
        <version>2.9.1</version>
    </dependency>
</dependencies>

To get started with the agent in your application, see the official documentation.

Sample App

To see the Agent in action first, checkout the xray-agent branch of the eb-java-scorekeep. The README has steps to set up a fully-functioning Spring Boot application instrumented with the X-Ray Agent and complemented with manual instrumentation by the X-Ray SDK.

Performance impacts

To get an idea of how much impact the X-Ray Agent might have on your system, please see the benchmarking package.

Customizing the Agent

Configuration

The X-Ray Agent is configured by an external, user-provided JSON file. By default this file is expected to be located at the root of the user's classpath and titled xray-agent.json. You can configure a custom location for the config file by setting the com.amazonaws.xray.configFile system property to the absolute filesystem path OR absolute location on the classpath of your configuration file.

For more details about configuring the agent, see the official documentation.

Toggling Tracing for Different Events

The benefit of the X-Ray Agent operating as a plugin rather than a monolith is that you can add or remove other compatible DiSCo plugins as you please. All of the plugins that the X-Ray agent will use are in the directory pointed to by the pluginPath parameter in your JVM argument. The plugins are what instrument different events like AWS SDK requests and SQL queries so they can be traced by X-Ray. Normally, this is the disco-plugins directory in the ZIP distribution, but it can be any directory.

To remove an undesirable plugin, simply remove it from your pluginPath directory and restart your app. For example, to disable tracing of SQL queries, remove the disco-java-agent-sql-plugin from your pluginPath.

To add new DiSCo plugins, just add them as a JAR to your pluginPath directory and restart your app. For example, if a plugin to intercept HTTP requests made with OkHTTP was developed, you should be able to just build the plugin and add the JAR to your pluginPath. No additional configuration from the agent should be required, but feel free to open an issue if it doesn't work out of the box.

Troubleshooting

When troubleshooting the agent, one of the first steps is to enable logging for the agent. The agent uses Apache Commons Logging, so you may need to add a bridge like log4j-jcl or jcl-over-slf4j to your classpath to see logs. To configure the log level, set this system property:

logging.level.com.amazonaws.xray = DEBUG

If your problem cannot be diagnosed with X-Ray logs, then DiSCo logs could help too. To enable DiSCo logs, append the classpath of the DiSCo logger to your JVM argument that's enabling the X-Ray agent, like so:

-javaagent:/<path-to-disco>/disco-java-agent.jar=pluginPath=/<path-to-disco>/disco-plugins:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose

For more troubleshooting steps, see the official documentation.

Developing on the Agent

Structure of this repo

This repository contains the X-Ray Agent as a DiSCo plugin. Note that this is NOT a proper Java agent with a premain and bytecode manipulation. Rather it is a plugin to extend the functionality of a proper Java agent like the one described. To learn more about DiSCo plugins and how they work with the DiSCo java agent, see the DiSCo documentation.

The layout of this project is:

aws-xray-agent - The source code of the AWS X-Ray agent plugin. This contains the hooks that allow our plugin to communicate with the DiSCo Agent. It is also where instrumentation using the X-Ray SDK happens.

aws-xray-agent-plugin - This package contains no source code. It only uses a series of build rules to bundle the above source code into a JAR that represents a DiSCo plugin, run integration tests against that JAR, then finally bundle that JAR and all needed DiSCo dependencies into an archive for the end user.

aws-xray-agent-benchmark - This package also contains no source code. It runs tests to compare the performance of the X-Ray Agent and the X-Ray SDK.

Building from Source

If there are unreleased changes on the main branch that you'd like to try out early, you can build the agent from its source code. The agent uses Gradle to manage its builds and produce the xray-agent.zip artifact that is ultimately distributed with releases. You can build the agent distribution locally by running the following commands:

git clone https://github.com/aws/aws-xray-java-agent.git
cd aws-xray-java-agent/
./gradlew build

Now, the latest changes on main will be bundled into a ZIP file located at aws-xray-agent-plugin/build/dist/xray-agent.zip. This ZIP file is structured the same as the one described in the installation documentation, so you can follow those instructions using this artifact. For example, if you'd like to extract the X-Ray Agent JAR and its dependencies to use in your project, you could run the following commands:

cd aws-xray-agent-plugin/build/dist/
unzip xray-agent.zip                 # Unpackages Agent JAR and disco dependencies into a disco directory
cp -r disco /path/to/your/project    # Copies the disco directory for use in your project with the -javaagent argument

Getting Help

Please use these community resources for getting help.

License

The AWS X-Ray SDK Java Agent is licensed under the Apache 2.0 License. See LICENSE and NOTICE.txt for more information.

aws-xray-java-agent's People

Contributors

alivest avatar amazon-auto avatar awssandra avatar bhautikpip avatar c1tadel avatar chanchiem avatar dependabot[bot] avatar srprash avatar sullis avatar willarmiros avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aws-xray-java-agent's Issues

Context propagation issue across threads in Lambda

This might be an X-Ray issue as DiSCo does this all in a very low level. It may be our transaction context resolver isn't properly detecting the facade segment in new threads. More investigation needed.

Reproduction

Subsegment subsegment = AWSXRay.beginSubsegment("handleRequest");
Future<Void> eventFuture = sendEvent(locationId);
...
eventFuture.get();
...
AWSXRay.endSubsegment();
private Future<Void> sendEvent(String message) {
    return executor.submit(() -> {
        Subsegment subsegment = AWSXRay.beginSubsegment("sendEvent");
        try {
            sns.publish(System.getenv("MAINTENANCE_TOPIC_ARN"), message);
            return null;
        }
        finally {
            AWSXRay.endSubsegment();
        }
    });
}

Support using the Agent as a layer on AWS Lambda

When adding the AWS X-Ray Java Agent to a Lambda function via a layer it causes cold starts to be much (30+ seconds) slower. This increase is so dramatic that it is a blocker for having official Lambda support. This is a result of the initialization work the agent must do to auto-instrument your Lambda code, and appears to be a problem with all auto-instrumentation java agents on the Lambda platform. An investigation is needed with the Lambda team to resolve this.

Add support for Play Framework

Play Framework is a Java web framework that does not use servlets but is often used in multi-threaded applications. Therefore it would be a good fit to be auto-instrumented by the agent. We would likely need to write a new Disco plugin to add such support.

Wildfly agent issue

I'm trying to run the disco agent with wildfly 19.0.0. It's acting as if if doesn't know where the disco-java-agent.jar is, but it obviously found it to initialize X-Ray.

Picked up JAVA_TOOL_OPTIONS: -javaagent:/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-java-agent.jar=pluginPath=/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-plugins
Feb 19, 2021 11:52:45 PM com.amazonaws.xray.agent.runtime.config.XRaySDKConfiguration init
INFO: Initializing the X-Ray Agent Recorder
Feb 19, 2021 11:52:49 PM org.jboss.msc.service.ServiceContainerImpl <clinit>
INFO: JBoss MSC version 1.4.11.Final
Feb 19, 2021 11:52:49 PM org.jboss.threads.Version <clinit>
INFO: JBoss Threads version 2.3.3.Final
java.lang.NoClassDefFoundError: software/amazon/disco/agent/concurrent/ExecutorInterceptor$ExecuteAdvice
        at [email protected]//org.jboss.msc.service.ServiceContainerImpl$ContainerExecutor.execute(ServiceContainerImpl.java:945)
        at [email protected]//org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:795)
        at [email protected]//org.jboss.msc.service.ServiceControllerImpl.commitInstallation(ServiceControllerImpl.java:313)
        at [email protected]//org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:752)
        at [email protected]//org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:260)
        at [email protected]//org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:192)
        at [email protected]//org.jboss.as.controller.ControlledProcessStateService.addService(ControlledProcessStateService.java:59)
        at [email protected]//org.jboss.as.server.BootstrapImpl.internalBootstrap(BootstrapImpl.java:118)
        at [email protected]//org.jboss.as.server.BootstrapImpl.bootstrap(BootstrapImpl.java:80)
        at [email protected]//org.jboss.as.server.Main.main(Main.java:109)
        at org.jboss.modules.Module.run(Module.java:352)
        at org.jboss.modules.Module.run(Module.java:320)
        at org.jboss.modules.Main.main(Main.java:593)
Feb 19, 2021 11:52:49 PM org.jboss.as.server.SystemExiter$1 logExit
FATAL: WFLYSRV0239: Aborting with exit code 1

Any idea what might be going wrong?

Thanks!

Downstream AWS service calls annotated as remote

I have just downloaded and tested v2.8.0. I previously had the issue mentioned in issue 64, but that seems to have been resolved. I can see all nodes I expect in the service map now, however the AWS service downstream calls don't look like I would expect.

Normally when you call for example a dynamo DB table, you will see the table name, and type as AWS::DynamoDB::Table.
But for dynamodb tables my nodes just have name: "dynamodb.eu-west-1.amazonaws.com" and type: "remote". For SSM calls I just see name: "ssm.eu-west-1.amazonaws.com", type: "remote" etc.

The only node that seems to have the correct annotation besides HTTP calls that actually are remote calls, is the container where the calls originate from, it has the type: "AWS::ECS::Container"

I'm not sure if this is related to the fix or not, but I haven't seen this before.
image
image

AWS Xray Java auto instrumentation using Agent Jar not recording all the requests

I have configured the AWS Xary using Java agent in ECS. But Xray automatic instrumentation using java agent only records the health check requests. All the requests which are going through ALB is not getting recognized by the Java agent. There is no sampling rule set to restrict the requests coming from ALB. I have tried to hit the service directly using the container IP address instead of ALB URL and that requests are getting recognized by the Xray Agent.

Following is the container definition for Xray daemon.
{
"Name": "xray-daemon",
"Image": {"Ref": "XrayImageURL01"},
"PortMappings" : [
{
"hostPort": 0,
"containerPort": 2000,
"protocol": "udp"
}
],
"LogConfiguration": {
"LogDriver": "awslogs",
"Options": {
"awslogs-group": {
"Ref": "ContainerCWLogGroup01"
},
"awslogs-region": { "Ref": "AWS::Region" },
"awslogs-stream-prefix": {"Fn::Sub": "${MicroserviceId01}"}
}
}
}

The agent jar is placed inside the container and following is the entry point script to add the java agent while starting the application.java -javaagent:/usr/app/disco/disco-java-agent.jar=pluginPath=/usr/app/disco/disco-plugins:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose -jar test-0.0.1-SNAPSHOT.jar

No other code level changes are applied for XRay in the application which is deployed in ECS.

Support Spring services

Currently there exists a race condition between the Java Agent and Spring when the agent is being bootstrap loaded. This is a known issue and being actively worked on so we can support auto-instrumentation of Spring applications as soon as possible.

X-Ray Errors

I'm going to create an AWS Support case too but just sharing with the agent team as there might be something here 🤷

I'm betting this is something with the underlying X-Ray SDK/Service but calling it out in hopes that somebody can point me in a direction as to how to resolve the issue. We added the X-Ray Java Agent a few weeks ago and it's been running great. But the last 2 days we've seen the following error 11 times in our production account:

com.amazonaws.xray.internal.XrayClientException: Could not read response code.
	at com.amazonaws.xray.internal.UnsignedXrayClient.sendRequest(UnsignedXrayClient.java:149)
	at com.amazonaws.xray.internal.UnsignedXrayClient.getSamplingTargets(UnsignedXrayClient.java:116)
	at com.amazonaws.xray.strategy.sampling.pollers.TargetPoller.pollManifest(TargetPoller.java:101)
	at com.amazonaws.xray.strategy.sampling.pollers.TargetPoller.lambda$start$0(TargetPoller.java:67)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at software.amazon.disco.agent.concurrent.decorate.DecoratedRunnableScheduledFuture.run(DecoratedRunnableScheduledFuture.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at software.amazon.disco.agent.concurrent.decorate.DecoratedRunnable.run(DecoratedRunnable.java:60)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:754)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1615)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
	at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
	at com.amazonaws.xray.internal.UnsignedXrayClient.sendRequest(UnsignedXrayClient.java:147)
	... 11 common frames omitted

Slow application start with xray agent

I recently added aws-xray-java-agent 2.7.1 to an app running Spring boot 2.3.5.

Before I added the agent, startup typically took around 3-4 seconds.
Started Application in 3.927 seconds (JVM running for 4.282)

Using aws-xray-java-agent 2.7.1, the startup of my application takes anywhere between 5-10 times longer.

Basically put the disco jars in place and added this line to my gradle.build

bootRun {
    jvmArgs = ["-javaagent:disco/disco-java-agent.jar=pluginPath=disco/disco-plugins"]
}

Started Application in 21.268 seconds (JVM running for 23.917)

Does this slowdown seem reasonably, or is something going on with my setup?

executing list all objects S3 API with Java agent throws exception

Description:

Executing list all S3 Objects API with Java agent throws exception right before sending the segment. I have attached code for reproduction .

import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.ListBucketsRequest;
import software.amazon.awssdk.services.s3.model.ListBucketsResponse;

S3Client s3 = S3Client.builder().region(region).build();

// List buckets
ListBucketsRequest listBucketsRequest = ListBucketsRequest.builder().build();
ListBucketsResponse listBucketsResponse = s3.listBuckets(listBucketsRequest);
listBucketsResponse.buckets().stream().forEach(x -> System.out.println(x.name()));

Maven Dependency:

<dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>s3</artifactId>
      <version>2.10.24</version>
</dependency>

Add Spring Boot Smoke Test

Spring Boot packages its apps in a unique way, and loads dependencies using a custom ClassLoader that makes them invisble to the System Class Loader which the agent uses at premain to discover useful dependencies like Trace ID injection libraries. More generally, we've had issues specific to spring boot in the past and should have a way to smoke test Spring Boot apps.

See OTel Java Instrumentation Spring Boot smoke tests for reference: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/67371ecfc9fc9b4cf7985ae96ffc5bde9a246b11/smoke-tests/springboot

Java agent interferes with Spring Boot executable jar classpath resolution

I have a trivial sample Spring Boot application that I'm attempting to run as an executable jar. The app runs fine without the agent with Jetty and Tomcat. The app also runs fine with the agent when started via Maven. However, starting the executable jar with the documented instructions for including an agent results in the error java.lang.NoClassDefFoundError: javax/servlet/Filter.

This class is definitely available:

[ssm-user@ip-10-0-146-244 merlin]$ unzip -l target/merlin-svc.war | grep javax
    95806  03-17-2023 19:36   WEB-INF/lib/javax.servlet-api-3.1.0.jar
    26586  03-17-2023 19:36   WEB-INF/lib/javax.annotation-api-1.3.2.jar

Working invocations:

./mvnw spring-boot:run -Dspring-boot.run.jvmArguments="-javaagent:/opt/aws/disco/disco-java-agent.jar=pluginPath=/opt/aws/disco/disco-plugins/"

# This only works with the Netty branch
java -jar merlin-svc.war

Failing invocation:

# Same behavior with and without verbose logging, see attached log below
java -javaagent:/opt/aws/disco/disco-java-agent.jar=pluginPath=/opt/aws/disco/disco-plugins/:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose -jar /opt/xxxx/merlin/merlin-svc.war

disco.log

Not emitting segment named 'demo' as it parents in-progress subsegments

While implementing the xray java agent i noticed an issue with multithreaded code. My application (API) builds a large response which consist of multiple content blocks. Each content block makes one or more request to multiple services. The creation of a block is wrapped in a java.util.concurrent.Future.

The aws x ray agent is not always closing the subsegments which are created for the outgoing apache client calls. Because of this the parent segment is never send since it has in-progress subsegments. However the outgoing calls/segments all have been finished this is clearly visible when debugging. I created a really basic demo application which also shows this behaviour. This simplified demo directly calls the feignclient while the actual application uses Resilience4j to wrap the feign calls. The agent has no trouble closing subsegments which are wrapped by Resilience4j and also run in a separate thread.

The first time http://localhost:8080/posts is called it seems to actually close all the subsegments and sends the udp packet to the deamon however when you make another call to http://localhost:8080/posts it fails. You can tweak how many external calls are made by tweaking the IntStream.range(1, 10).

Here you can see that even though the first request seems ok it actually thinks most of the sub segments are still in progress.
Image

https://github.com/wesleymooiman/xray-agent-demo (https://github.com/wesleymooiman/xray-agent-demo/blob/master/src/main/java/com/example/demo/post/PostController.java)

2020-12-18 12:10:04,791 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5351c1c1cb2f1b10 DEBUG         TargetPoller:125  - Applicable default rule: Default
2020-12-18 12:10:04,791 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5351c1c1cb2f1b10 DEBUG      CentralizedRule:202  - Sampling target has expired for rule Default. Borrowing a request.
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,836 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@03a17fae4a223484 DEBUG XRayTransactionContext:79   - Ending subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,836 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@19c5b115e7e65c92 DEBUG XRayTransactionContext:79   - Ending subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,836 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@72ae23b1caa741dd DEBUG XRayTransactionContext:79   - Ending subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,836 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@03a17fae4a223484 DEBUG       SubsegmentImpl:52   - Subsegment named 'jsonplaceholder.typicode.com' ending. Parent segment named 'demo' has reference count 7
2020-12-18 12:10:04,836 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@72ae23b1caa741dd DEBUG       SubsegmentImpl:52   - Subsegment named 'jsonplaceholder.typicode.com' ending. Parent segment named 'demo' has reference count 7
2020-12-18 12:10:04,836 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@19c5b115e7e65c92 DEBUG       SubsegmentImpl:52   - Subsegment named 'jsonplaceholder.typicode.com' ending. Parent segment named 'demo' has reference count 7
2020-12-18 12:10:04,843 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5351c1c1cb2f1b10 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,844 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5351c1c1cb2f1b10 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,848 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5209a45bf387f937 DEBUG XRayTransactionContext:79   - Ending subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,848 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5209a45bf387f937 DEBUG       SubsegmentImpl:52   - Subsegment named 'jsonplaceholder.typicode.com' ending. Parent segment named 'demo' has reference count 6
2020-12-18 12:10:04,876 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5cb887f741db0ba5 DEBUG XRayTransactionContext:79   - Ending subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,877 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5cb887f741db0ba5 DEBUG       SubsegmentImpl:52   - Subsegment named 'jsonplaceholder.typicode.com' ending. Parent segment named 'demo' has reference count 5
2020-12-18 12:10:04,890 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5351c1c1cb2f1b10 DEBUG      AWSXRayRecorder:539  - Ending segment named 'demo'.
2020-12-18 12:10:04,890 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5351c1c1cb2f1b10 DEBUG      AWSXRayRecorder:549  - Not emitting segment named 'demo' as it parents in-progress subsegments.

Dynamic name using only disco library

I'm using disco for spring.
Is it possible to specify the dynamic name of disco using YML profile properties or Java config?

Example
Default Name : XRayInstrumentedService
Dynamic Name : Production??

I couldn't find it. But it seemed like there was, so I asked a question

Thank you for checking.

Jackson dependency & Shadow

Hello there, we're seeing an issue with our Springboot app where the jackson depedancy is being changed by the agent and causing our app to fail.

I'm aware from looking through the code that com.github.johnrengelman.shadow is used to prevent this but I am not familiar enough with gradle to know how this has been configured in relation to the jackson library.

Any guidance would be appreciated.

I hope disco works with javaagent attach method

Hello
Sorry to open the issue again

The method that javaagent is started when the app is started executes the premain service logic, but the attach method does not work normally because it enters other service logic while the application is running
Supporting the attach method makes the programming method more beautiful and free
Please review positively. Thank

Lambda Installer should not require a service name

The agent installer used in lambda is invoked similar to the following way:
XRayAgentInstaller.installInLambda("servicename=MyLambdaFunction")

In Lambda, the segment is generated on behalf of the function code by the container. The function code retrieves this segment through a concept called a Facade Segment, an immutable segment. Because the service name is never even used, the lambda installer should by default not need a service name parameter.

XRAY Agent Application Logging

We use the java auto-instrumentation agent to capture traces for our applications in fargate.
However, in our application logs we are seeing a lot of verbosity coming from the agent (below).

Is there any way we can suppress/turn-off the logging, or send it to another CloudWatch log so that the application logs are not contaminated. The XRAY auto-instrumentation agent is supposed to be a non-invasive/non-intrusive tool to monitoring applications so we are looking for either a system property similar to
logging.level.com.amazonaws.xray or a switch in the xray-agent.json config file.

The auto-instrumentation java agent is a great solution because its not possible for us to instrument the underlying application code using the SDK. XRAY is critical to our overall monitoring strategy because we are attempting to remove DataDog in favour of AWS native monitoring solutions and to reduce our overall costs running the estate.

26-Mar-2021 06:08:04.669 SEVERE [kyc-quartz-scheduler_Worker-5] com.amazonaws.xray.agent.runtime.listeners.XRayListener.listen The X-Ray Agent had encountered an unexpected exception for the following event: software.amazon.disco.agent.event.ServiceDownstreamResponseEvent@2c0dc96f
java.lang.NullPointerException
at com.amazonaws.xray.agent.runtime.handlers.downstream.SqlHandler.handleResponse(SqlHandler.java:80)
at com.amazonaws.xray.agent.runtime.dispatcher.EventDispatcher.dispatchResponseEvent(EventDispatcher.java:75)
at com.amazonaws.xray.agent.runtime.listeners.XRayListener.listen(XRayListener.java:38)
at software.amazon.disco.agent.event.EventBus.publish(EventBus.java:66)

xray_application_log.txt

Using xray-agent 3.3.3 in AML2 on tomcat server

i have added the agent in classpath and also plugins. but we are unablke to see SQL Traces. Please let me know if there is any configuration file that we need to enable like xray-agent.json to view the SQL traces and examine them. We are able to see http requests but not SQL Queries.

X-ray is getting initialized by below

CATALINA_OPTS='-Xms512M -Xmx1G -javaagent:/path to disco/disco-java-agent.jar=pluginPath=/path to disco/disco-plugins'

ECS Fargate Sidecar: X-Ray throws warnings at startup

I'm deploying my application as a container that's deployed to ECS Fargate w/X-Ray Daemon as a sidecar. All of the expected traces are showing up in X-Ray but my OCD is pained by the WARN logs that show up on startup 😂

At the startup of my application I get the following WARN logs:

java.net.ConnectException: Invalid argument (connect failed)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
	at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
	at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
	at com.amazonaws.xray.plugins.EC2MetadataFetcher.fetchString(EC2MetadataFetcher.java:159)
	at com.amazonaws.xray.plugins.EC2MetadataFetcher.fetchToken(EC2MetadataFetcher.java:123)
	at com.amazonaws.xray.plugins.EC2MetadataFetcher.fetch(EC2MetadataFetcher.java:72)
	at com.amazonaws.xray.plugins.EC2Plugin.<init>(EC2Plugin.java:72)
	at com.amazonaws.xray.plugins.EC2Plugin.<init>(EC2Plugin.java:67)
	at com.amazonaws.xray.AWSXRayRecorderBuilder.withDefaultPlugins(AWSXRayRecorderBuilder.java:216)
	at com.amazonaws.xray.agent.runtime.config.XRaySDKConfiguration.init(XRaySDKConfiguration.java:204)
	at com.amazonaws.xray.agent.runtime.config.XRaySDKConfiguration.init(XRaySDKConfiguration.java:158)
	at com.amazonaws.xray.agent.runtime.AgentRuntimeLoader.configureXRay(AgentRuntimeLoader.java:74)
	at com.amazonaws.xray.agent.runtime.AgentRuntimeLoader.init(AgentRuntimeLoader.java:53)
	at com.amazonaws.xray.agent.runtime.AgentRuntimeLoader.init(AgentRuntimeLoader.java:37)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at software.amazon.disco.agent.plugin.PluginDiscovery.apply(PluginDiscovery.java:172)
	at software.amazon.disco.agent.DiscoAgentTemplate.install(DiscoAgentTemplate.java:145)
	at software.amazon.disco.agent.DiscoAgentTemplate.install(DiscoAgentTemplate.java:107)
	at software.amazon.disco.agent.DiscoAgent.impl(DiscoAgent.java:91)
	at software.amazon.disco.agent.DiscoAgent.premain(DiscoAgent.java:52)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)```


FR: Support for Spring WebClient

FR: Support for Spring WebClient

In my sample application using Spring WebClient with X-Ray Java agent auto-instrumentation, the trace header is not injected into the http header.

This is the code block for the downstream http call using Spring webclient

import org.springframework.web.reactive.function.client.WebClient;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequiredArgsConstructor
@Slf4j
public class GoodbyeController {

  private final WebClient additionWebClient;

  @GetMapping("/goodbye")
  public ResponseEntity<?> goodbye() throws InterruptedException {

      MathResult addResult =
          this.additionWebClient
              .get()
              .uri(
                  uri ->
                      uri.path("/add")
                          .queryParam("x", Long.toString(x))
                          .queryParam("y", Long.toString(y))
                          .build())
              .retrieve()
              .bodyToMono(MathResult.class)
              .block();
   }
}

In the downstream http request handler, I printed out the request header and found that no trace id,

2021-03-04 16:19:18.841 INFO [Addition App,604108865e2cf420e5d967d9c6dc0ec3,e5d967d9c6dc0ec3,true] 25 --- [nio-8080-exec-8] c.g.t.s.tracingpoc.AdditionController : Starting to add... x: [100], y: [1000]. RequestHeaders: [{accept-encoding=gzip, user-agent=ReactorNetty/0.9.8.RELEASE, host=<redacted>:8080, accept=*/*}]

Agent doesn't take X-Forwarded-For header into account

We keep seeing client_ip field set as 127.0.0.1 even though X-Forwarded-For header is set in the request.
From X-Ray Concepts:

If a load balancer or other intermediary forwards a request to your application, X-Ray takes the client IP from the X-Forwarded-For header in the request instead of from the source IP in the IP packet.

I wonder if it's due to the use of getLocalIPAddress() method here

    private void populateHeaderToTransactionState(HttpNetworkProtocolRequestEvent requestEvent, XRayTransactionState transactionState) {
        transactionState.withHost(requestEvent.getHost())
                .withMethod(requestEvent.getMethod())
                .withUrl(requestEvent.getURL())
                .withUserAgent(requestEvent.getUserAgent())
                .withClientIP(requestEvent.getLocalIPAddress())
                .withTraceheaderString(requestEvent.getHeaderData(HEADER_KEY));

When Disco is published to MVN Central, remove local import

PR #28 adds a workaround to allow Travis CI builds to be executed fully. This workaround imports Disco locally in the project so that Travis CI can compile the agent code without requiring Disco to be published into Maven Central.

Once Disco has been published to Maven Central, these lines should be removed, as well as the disco-lib folders contained in the following subprojects:

  • aws-xray-auto-instrumentation-agent-bootstrap
  • aws-xray-auto-instrumentation-agent-installer

Trace ID injection doesn't work with Spring Boot

Enabling Trace ID injection via the agent is not currently possible when using Spring Boot. The agent relies on the aws-xray-recorder-sdk-slf4j dependency to be available on the classpath when the agent starts up so it can add the trace ID injector. When the Spring Boot app is packaged with mvn clean package, Spring Boot does packages dependencies weirdly in a way that's not visible to the system class loader when the Agent is starting up.

I think the solution here is to lazy load the trace ID injector, after the spring boot app has started up, once the dependency is discoverable.

Xray event emitting does not limit UDP packet size

Hello folks, it looks like the UDP event emitter doesn't not limit the UDP packet size causing these annoying errors:

java.io.IOException: Message too long
at java.base/java.net.PlainDatagramSocketImpl.send(Native Method)
at java.base/java.net.DatagramSocket.send(DatagramSocket.java:695)
at com.amazonaws.xray.emitters.UDPEmitter.sendData(UDPEmitter.java:106)
at com.amazonaws.xray.emitters.UDPEmitter.sendSegment(UDPEmitter.java:84)
at com.amazonaws.xray.AWSXRayRecorder.sendSegment(AWSXRayRecorder.java:178)
at com.amazonaws.xray.contexts.ThreadLocalSegmentContext.endSubsegment(ThreadLocalSegmentContext.java:77)
at com.amazonaws.xray.AWSXRayRecorder.endSubsegment(AWSXRayRecorder.java:546)
at com.amazonaws.xray.AWSXRay.endSubsegment(AWSXRay.java:111)
at com.amazonaws.xray.sql.mysql.TracingInterceptor$TracingStatementProxy.invoke(TracingInterceptor.java:198)
at com.sun.proxy.$Proxy64.execute(Unknown Source)
at jdk.internal.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114)
at com.sun.proxy.$Proxy64.execute(Unknown Source)
at org.jdbi.v3.core.statement.SqlLoggerUtil.wrap(SqlLoggerUtil.java:31)
at org.jdbi.v3.core.statement.SqlStatement.internalExecute(SqlStatement.java:1666)
at org.jdbi.v3.core.result.ResultProducers.lambda$returningUpdateCount$0(ResultProducers.java:49)
at org.jdbi.v3.core.statement.Update.execute(Update.java:53)
at org.jdbi.v3.core.statement.Update.execute(Update.java:41)
at com.scanii.core.models.daos.DistributedLockDao.lambda$acquire$0(DistributedLockDao.java:55)
at org.jdbi.v3.core.transaction.LocalTransactionHandler.inTransaction(LocalTransactionHandler.java:134)
at org.jdbi.v3.core.transaction.LocalTransactionHandler.inTransaction(LocalTransactionHandler.java:158)
at org.jdbi.v3.core.Handle.inTransaction(Handle.java:467)
[...]

According to wikipedia (tm), the datagram packet size limit is 64kb (https://en.wikipedia.org/wiki/User_Datagram_Protocol) and yes, there's no good reason for the trace to be that large, but, in any case, it probably makes sense for xray to discard the trace instead of trying so send the packet and throw an exception.

Using X-Ray Agent in Spring Boot & Zuul Proxy API Gateway

We are trying to instrument our microservices using x-ray java agent. Our microservices are implemented using Spring Boot framework. For other microservices, we accomplished the auto instrumentation using the disco plugins by adding environment variables to our applications as below.

-Dcom.amazonaws.xray.strategy.tracingName=serviceName
-Dcom.amazonaws.xray.strategy.contextMissingStrategy=IGNORE_ERROR
-Dcom.amazonaws.xray.emitter.daemonAddress=127.0.0.1:2000
-Dcom.amazonaws.xray.tracingEnabled=True
-javaagent:src/main/resources/disco/disco-java-agent.jar=pluginPath=src/main/resources/disco/disco-plugins

But in our API Gateway, we failed to do so. While we try to trace any request from our API Gateway, we came across an error like this below:

{
"error": "Exception while serializing entity.",
"stack_trace":"com.fasterxml.jackson.databind.JsonMappingException: Failed to load class
'com.amazonaws.xray.entities.SegmentImpl$Access4JacksonSerializerdb8acfce':
com.fasterxml.jackson.module.afterburner.ser.BeanPropertyAccessor
com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:295)
}
full-stack-trace.json.zip

Also in our x-ray daemon running locally, we encountered with this warning below:

[Warn] Missing header or segment: {"format": "json", "version": 1}

To resolve this error, we tried to exclude plugin path from command like below:

-Dcom.amazonaws.xray.strategy.tracingName=bid-service-local
-Dcom.amazonaws.xray.strategy.contextMissingStrategy=IGNORE_ERROR
-Dcom.amazonaws.xray.emitter.daemonAddress=127.0.0.1:2000
-Dcom.amazonaws.xray.tracingEnabled=True
-javaagent:src/main/resources/disco/disco-java-agent.jar

But there was no segment sent to the daemon that is running locally. Then we tried to add the plugins to its path one by one,

aws-xray-agent-plugin.jar
disco-java-agent-aws-plugin.jar

First, we added the two jars above to disco-plugins path above, there was no error, but segment also not sent to our daemon .

Then, we added the jar below and, we saw that this error is related to this plugin, with some dependencies that we use in our API Gateway application.

disco-java-agent-web-plugin.jar

Dependencies & Technologies

Spring Boot version is 2.3.8.

 - implementation "org.springframework.boot:spring-boot-configuration-processor"
 - implementation "org.springframework.boot:spring-boot-starter-actuator"
 - implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
 - implementation 'org.springframework.cloud:spring-cloud-starter-netflix-zuul'
 - implementation 'org.springframework.cloud:spring-cloud-starter-netflix-hystrix-dashboard'
 - implementation 'org.springframework.cloud:spring-cloud-starter-netflix-hystrix'
 - implementation 'org.springframework.cloud:spring-cloud-starter-netflix-ribbon'
 - implementation "org.springframework.cloud:spring-cloud-starter-sleuth:2.2.8.RELEASE"

Also we are using ZuulFilter to add custom headers to incoming requests' header. Could this also be an issue for our error that we encounter, how can we move forward from here? 🙏

Add Disco Locally

Since Disco has not yet been published to Maven Central, we need to pull it in locally ourselves so that we can start running Travis CI tests.

Update Java SDK and add aws-java-sdk-sts library inside jar

Hi,

Currently we are implementing the AWS X-Ray auto-instrumentation agent for Java for a Spring project inside EKS, but we had problems executing the spring microservices for two reasons:

  • The version of the sdk we are using is more recent than the java agent (1.11.909)
  • In the EKS, the pods are using serviceaccount to call aws resources, for this, the java agent needs to have the aws-java-sdk-sts library in its classpath.

We hope you can update the aws-xray-java-agent with those changes :)

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.