aws / aws-xray-java-agent Goto Github PK

I'm deploying my application as a container that's deployed to ECS Fargate w/X-Ray Daemon as a sidecar. All of the expected traces are showing up in X-Ray but my OCD is pained by the WARN logs that show up on startup 😂

At the startup of my application I get the following WARN logs:

java.net.ConnectException: Invalid argument (connect failed)
	at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
	at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:474)
	at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:569)
	at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:341)
	at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:362)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1592)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
	at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
	at com.amazonaws.xray.plugins.EC2MetadataFetcher.fetchString(EC2MetadataFetcher.java:159)
	at com.amazonaws.xray.plugins.EC2MetadataFetcher.fetchToken(EC2MetadataFetcher.java:123)
	at com.amazonaws.xray.plugins.EC2MetadataFetcher.fetch(EC2MetadataFetcher.java:72)
	at com.amazonaws.xray.plugins.EC2Plugin.<init>(EC2Plugin.java:72)
	at com.amazonaws.xray.plugins.EC2Plugin.<init>(EC2Plugin.java:67)
	at com.amazonaws.xray.AWSXRayRecorderBuilder.withDefaultPlugins(AWSXRayRecorderBuilder.java:216)
	at com.amazonaws.xray.agent.runtime.config.XRaySDKConfiguration.init(XRaySDKConfiguration.java:204)
	at com.amazonaws.xray.agent.runtime.config.XRaySDKConfiguration.init(XRaySDKConfiguration.java:158)
	at com.amazonaws.xray.agent.runtime.AgentRuntimeLoader.configureXRay(AgentRuntimeLoader.java:74)
	at com.amazonaws.xray.agent.runtime.AgentRuntimeLoader.init(AgentRuntimeLoader.java:53)
	at com.amazonaws.xray.agent.runtime.AgentRuntimeLoader.init(AgentRuntimeLoader.java:37)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at software.amazon.disco.agent.plugin.PluginDiscovery.apply(PluginDiscovery.java:172)
	at software.amazon.disco.agent.DiscoAgentTemplate.install(DiscoAgentTemplate.java:145)
	at software.amazon.disco.agent.DiscoAgentTemplate.install(DiscoAgentTemplate.java:107)
	at software.amazon.disco.agent.DiscoAgent.impl(DiscoAgent.java:91)
	at software.amazon.disco.agent.DiscoAgent.premain(DiscoAgent.java:52)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)```

I'm going to create an AWS Support case too but just sharing with the agent team as there might be something here 🤷

I'm betting this is something with the underlying X-Ray SDK/Service but calling it out in hopes that somebody can point me in a direction as to how to resolve the issue. We added the X-Ray Java Agent a few weeks ago and it's been running great. But the last 2 days we've seen the following error 11 times in our production account:

com.amazonaws.xray.internal.XrayClientException: Could not read response code.
	at com.amazonaws.xray.internal.UnsignedXrayClient.sendRequest(UnsignedXrayClient.java:149)
	at com.amazonaws.xray.internal.UnsignedXrayClient.getSamplingTargets(UnsignedXrayClient.java:116)
	at com.amazonaws.xray.strategy.sampling.pollers.TargetPoller.pollManifest(TargetPoller.java:101)
	at com.amazonaws.xray.strategy.sampling.pollers.TargetPoller.lambda$start$0(TargetPoller.java:67)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at software.amazon.disco.agent.concurrent.decorate.DecoratedRunnableScheduledFuture.run(DecoratedRunnableScheduledFuture.java:68)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at software.amazon.disco.agent.concurrent.decorate.DecoratedRunnable.run(DecoratedRunnable.java:60)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.base/java.net.SocketInputStream.socketRead0(Native Method)
	at java.base/java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:168)
	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
	at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252)
	at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:292)
	at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
	at java.base/sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:754)
	at java.base/sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:689)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1615)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1520)
	at java.base/java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:527)
	at com.amazonaws.xray.internal.UnsignedXrayClient.sendRequest(UnsignedXrayClient.java:147)
	... 11 common frames omitted

While implementing the xray java agent i noticed an issue with multithreaded code. My application (API) builds a large response which consist of multiple content blocks. Each content block makes one or more request to multiple services. The creation of a block is wrapped in a java.util.concurrent.Future.

The aws x ray agent is not always closing the subsegments which are created for the outgoing apache client calls. Because of this the parent segment is never send since it has in-progress subsegments. However the outgoing calls/segments all have been finished this is clearly visible when debugging. I created a really basic demo application which also shows this behaviour. This simplified demo directly calls the feignclient while the actual application uses Resilience4j to wrap the feign calls. The agent has no trouble closing subsegments which are wrapped by Resilience4j and also run in a separate thread.

The first time http://localhost:8080/posts is called it seems to actually close all the subsegments and sends the udp packet to the deamon however when you make another call to http://localhost:8080/posts it fails. You can tweak how many external calls are made by tweaking the IntStream.range(1, 10).

Here you can see that even though the first request seems ok it actually thinks most of the sub segments are still in progress.

https://github.com/wesleymooiman/xray-agent-demo (https://github.com/wesleymooiman/xray-agent-demo/blob/master/src/main/java/com/example/demo/post/PostController.java)

2020-12-18 12:10:04,791 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5351c1c1cb2f1b10 DEBUG         TargetPoller:125  - Applicable default rule: Default
2020-12-18 12:10:04,791 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5351c1c1cb2f1b10 DEBUG      CentralizedRule:202  - Sampling target has expired for rule Default. Borrowing a request.
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,792 1-5fdc8dff-7743171376d0ed885bf6d8af@6a9efe6eab992a14 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,836 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@03a17fae4a223484 DEBUG XRayTransactionContext:79   - Ending subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,836 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@19c5b115e7e65c92 DEBUG XRayTransactionContext:79   - Ending subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,836 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@72ae23b1caa741dd DEBUG XRayTransactionContext:79   - Ending subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,836 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@03a17fae4a223484 DEBUG       SubsegmentImpl:52   - Subsegment named 'jsonplaceholder.typicode.com' ending. Parent segment named 'demo' has reference count 7
2020-12-18 12:10:04,836 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@72ae23b1caa741dd DEBUG       SubsegmentImpl:52   - Subsegment named 'jsonplaceholder.typicode.com' ending. Parent segment named 'demo' has reference count 7
2020-12-18 12:10:04,836 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@19c5b115e7e65c92 DEBUG       SubsegmentImpl:52   - Subsegment named 'jsonplaceholder.typicode.com' ending. Parent segment named 'demo' has reference count 7
2020-12-18 12:10:04,843 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5351c1c1cb2f1b10 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,844 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5351c1c1cb2f1b10 DEBUG XRayTransactionContext:64   - Beginning subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,848 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5209a45bf387f937 DEBUG XRayTransactionContext:79   - Ending subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,848 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5209a45bf387f937 DEBUG       SubsegmentImpl:52   - Subsegment named 'jsonplaceholder.typicode.com' ending. Parent segment named 'demo' has reference count 6
2020-12-18 12:10:04,876 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5cb887f741db0ba5 DEBUG XRayTransactionContext:79   - Ending subsegment named: jsonplaceholder.typicode.com
2020-12-18 12:10:04,877 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5cb887f741db0ba5 DEBUG       SubsegmentImpl:52   - Subsegment named 'jsonplaceholder.typicode.com' ending. Parent segment named 'demo' has reference count 5
2020-12-18 12:10:04,890 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5351c1c1cb2f1b10 DEBUG      AWSXRayRecorder:539  - Ending segment named 'demo'.
2020-12-18 12:10:04,890 1-5fdc8e0c-dfcd49e48d80e11caabab3ad@5351c1c1cb2f1b10 DEBUG      AWSXRayRecorder:549  - Not emitting segment named 'demo' as it parents in-progress subsegments.

FR: Support for Spring WebClient

In my sample application using Spring WebClient with X-Ray Java agent auto-instrumentation, the trace header is not injected into the http header.

This is the code block for the downstream http call using Spring webclient

import org.springframework.web.reactive.function.client.WebClient;
import org.springframework.web.bind.annotation.RequestHeader;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
@RequiredArgsConstructor
@Slf4j
public class GoodbyeController {

  private final WebClient additionWebClient;

  @GetMapping("/goodbye")
  public ResponseEntity<?> goodbye() throws InterruptedException {

      MathResult addResult =
          this.additionWebClient
              .get()
              .uri(
                  uri ->
                      uri.path("/add")
                          .queryParam("x", Long.toString(x))
                          .queryParam("y", Long.toString(y))
                          .build())
              .retrieve()
              .bodyToMono(MathResult.class)
              .block();
   }
}

In the downstream http request handler, I printed out the request header and found that no trace id,

2021-03-04 16:19:18.841 INFO [Addition App,604108865e2cf420e5d967d9c6dc0ec3,e5d967d9c6dc0ec3,true] 25 --- [nio-8080-exec-8] c.g.t.s.tracingpoc.AdditionController : Starting to add... x: [100], y: [1000]. RequestHeaders: [{accept-encoding=gzip, user-agent=ReactorNetty/0.9.8.RELEASE, host=<redacted>:8080, accept=*/*}]

We use the java auto-instrumentation agent to capture traces for our applications in fargate.
However, in our application logs we are seeing a lot of verbosity coming from the agent (below).

Is there any way we can suppress/turn-off the logging, or send it to another CloudWatch log so that the application logs are not contaminated. The XRAY auto-instrumentation agent is supposed to be a non-invasive/non-intrusive tool to monitoring applications so we are looking for either a system property similar to
logging.level.com.amazonaws.xray or a switch in the xray-agent.json config file.

The auto-instrumentation java agent is a great solution because its not possible for us to instrument the underlying application code using the SDK. XRAY is critical to our overall monitoring strategy because we are attempting to remove DataDog in favour of AWS native monitoring solutions and to reduce our overall costs running the estate.

26-Mar-2021 06:08:04.669 SEVERE [kyc-quartz-scheduler_Worker-5] com.amazonaws.xray.agent.runtime.listeners.XRayListener.listen The X-Ray Agent had encountered an unexpected exception for the following event: software.amazon.disco.agent.event.ServiceDownstreamResponseEvent@2c0dc96f
java.lang.NullPointerException
at com.amazonaws.xray.agent.runtime.handlers.downstream.SqlHandler.handleResponse(SqlHandler.java:80)
at com.amazonaws.xray.agent.runtime.dispatcher.EventDispatcher.dispatchResponseEvent(EventDispatcher.java:75)
at com.amazonaws.xray.agent.runtime.listeners.XRayListener.listen(XRayListener.java:38)
at software.amazon.disco.agent.event.EventBus.publish(EventBus.java:66)

xray_application_log.txt

I have just downloaded and tested v2.8.0. I previously had the issue mentioned in issue 64, but that seems to have been resolved. I can see all nodes I expect in the service map now, however the AWS service downstream calls don't look like I would expect.

Normally when you call for example a dynamo DB table, you will see the table name, and type as AWS::DynamoDB::Table.
But for dynamodb tables my nodes just have name: "dynamodb.eu-west-1.amazonaws.com" and type: "remote". For SSM calls I just see name: "ssm.eu-west-1.amazonaws.com", type: "remote" etc.

The only node that seems to have the correct annotation besides HTTP calls that actually are remote calls, is the container where the calls originate from, it has the type: "AWS::ECS::Container"

I'm not sure if this is related to the fix or not, but I haven't seen this before.

Spring Boot packages its apps in a unique way, and loads dependencies using a custom ClassLoader that makes them invisble to the System Class Loader which the agent uses at premain to discover useful dependencies like Trace ID injection libraries. More generally, we've had issues specific to spring boot in the past and should have a way to smoke test Spring Boot apps.

See OTel Java Instrumentation Spring Boot smoke tests for reference: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/67371ecfc9fc9b4cf7985ae96ffc5bde9a246b11/smoke-tests/springboot

Hi,

Currently we are implementing the AWS X-Ray auto-instrumentation agent for Java for a Spring project inside EKS, but we had problems executing the spring microservices for two reasons:

• The version of the sdk we are using is more recent than the java agent (1.11.909)
• In the EKS, the pods are using serviceaccount to call aws resources, for this, the java agent needs to have the aws-java-sdk-sts library in its classpath.

We hope you can update the aws-xray-java-agent with those changes :)

Thanks!

When used with aws sdk the above crashes.
This can be solved by lowering the version of aws sdk, but I don't think this is the correct answer.

This might be an X-Ray issue as DiSCo does this all in a very low level. It may be our transaction context resolver isn't properly detecting the facade segment in new threads. More investigation needed.

Reproduction

Subsegment subsegment = AWSXRay.beginSubsegment("handleRequest");
Future<Void> eventFuture = sendEvent(locationId);
...
eventFuture.get();
...
AWSXRay.endSubsegment();

private Future<Void> sendEvent(String message) {
    return executor.submit(() -> {
        Subsegment subsegment = AWSXRay.beginSubsegment("sendEvent");
        try {
            sns.publish(System.getenv("MAINTENANCE_TOPIC_ARN"), message);
            return null;
        }
        finally {
            AWSXRay.endSubsegment();
        }
    });
}

Description:

Executing list all S3 Objects API with Java agent throws exception right before sending the segment. I have attached code for reproduction .

import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
import software.amazon.awssdk.services.s3.model.ListBucketsRequest;
import software.amazon.awssdk.services.s3.model.ListBucketsResponse;

S3Client s3 = S3Client.builder().region(region).build();

// List buckets
ListBucketsRequest listBucketsRequest = ListBucketsRequest.builder().build();
ListBucketsResponse listBucketsResponse = s3.listBuckets(listBucketsRequest);
listBucketsResponse.buckets().stream().forEach(x -> System.out.println(x.name()));

Maven Dependency:

<dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>s3</artifactId>
      <version>2.10.24</version>
</dependency>

I'm trying to run the disco agent with wildfly 19.0.0. It's acting as if if doesn't know where the disco-java-agent.jar is, but it obviously found it to initialize X-Ray.

Picked up JAVA_TOOL_OPTIONS: -javaagent:/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-java-agent.jar=pluginPath=/Users/monty/lark/goldfinch/wildfly-config/wildfly-modules/software/amazon/disco/main/disco-plugins
Feb 19, 2021 11:52:45 PM com.amazonaws.xray.agent.runtime.config.XRaySDKConfiguration init
INFO: Initializing the X-Ray Agent Recorder
Feb 19, 2021 11:52:49 PM org.jboss.msc.service.ServiceContainerImpl <clinit>
INFO: JBoss MSC version 1.4.11.Final
Feb 19, 2021 11:52:49 PM org.jboss.threads.Version <clinit>
INFO: JBoss Threads version 2.3.3.Final
java.lang.NoClassDefFoundError: software/amazon/disco/agent/concurrent/ExecutorInterceptor$ExecuteAdvice
        at [email protected]//org.jboss.msc.service.ServiceContainerImpl$ContainerExecutor.execute(ServiceContainerImpl.java:945)
        at [email protected]//org.jboss.msc.service.ServiceControllerImpl.doExecute(ServiceControllerImpl.java:795)
        at [email protected]//org.jboss.msc.service.ServiceControllerImpl.commitInstallation(ServiceControllerImpl.java:313)
        at [email protected]//org.jboss.msc.service.ServiceContainerImpl.install(ServiceContainerImpl.java:752)
        at [email protected]//org.jboss.msc.service.ServiceTargetImpl.install(ServiceTargetImpl.java:260)
        at [email protected]//org.jboss.msc.service.ServiceBuilderImpl.install(ServiceBuilderImpl.java:192)
        at [email protected]//org.jboss.as.controller.ControlledProcessStateService.addService(ControlledProcessStateService.java:59)
        at [email protected]//org.jboss.as.server.BootstrapImpl.internalBootstrap(BootstrapImpl.java:118)
        at [email protected]//org.jboss.as.server.BootstrapImpl.bootstrap(BootstrapImpl.java:80)
        at [email protected]//org.jboss.as.server.Main.main(Main.java:109)
        at org.jboss.modules.Module.run(Module.java:352)
        at org.jboss.modules.Module.run(Module.java:320)
        at org.jboss.modules.Main.main(Main.java:593)
Feb 19, 2021 11:52:49 PM org.jboss.as.server.SystemExiter$1 logExit
FATAL: WFLYSRV0239: Aborting with exit code 1

Any idea what might be going wrong?

Thanks!

Since Disco has not yet been published to Maven Central, we need to pull it in locally ourselves so that we can start running Travis CI tests.

The agent installer used in lambda is invoked similar to the following way:
XRayAgentInstaller.installInLambda("servicename=MyLambdaFunction")

In Lambda, the segment is generated on behalf of the function code by the container. The function code retrieves this segment through a concept called a Facade Segment, an immutable segment. Because the service name is never even used, the lambda installer should by default not need a service name parameter.

Play Framework is a Java web framework that does not use servlets but is often used in multi-threaded applications. Therefore it would be a good fit to be auto-instrumented by the agent. We would likely need to write a new Disco plugin to add such support.

I have configured the AWS Xary using Java agent in ECS. But Xray automatic instrumentation using java agent only records the health check requests. All the requests which are going through ALB is not getting recognized by the Java agent. There is no sampling rule set to restrict the requests coming from ALB. I have tried to hit the service directly using the container IP address instead of ALB URL and that requests are getting recognized by the Xray Agent.

Following is the container definition for Xray daemon.
{
"Name": "xray-daemon",
"Image": {"Ref": "XrayImageURL01"},
"PortMappings" : [
{
"hostPort": 0,
"containerPort": 2000,
"protocol": "udp"
}
],
"LogConfiguration": {
"LogDriver": "awslogs",
"Options": {
"awslogs-group": {
"Ref": "ContainerCWLogGroup01"
},
"awslogs-region": { "Ref": "AWS::Region" },
"awslogs-stream-prefix": {"Fn::Sub": "${MicroserviceId01}"}
}
}
}

The agent jar is placed inside the container and following is the entry point script to add the java agent while starting the application.java -javaagent:/usr/app/disco/disco-java-agent.jar=pluginPath=/usr/app/disco/disco-plugins:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose -jar test-0.0.1-SNAPSHOT.jar

No other code level changes are applied for XRay in the application which is deployed in ECS.

Hello there, we're seeing an issue with our Springboot app where the jackson depedancy is being changed by the agent and causing our app to fail.

I'm aware from looking through the code that com.github.johnrengelman.shadow is used to prevent this but I am not familiar enough with gradle to know how this has been configured in relation to the jackson library.

Any guidance would be appreciated.

We keep seeing client_ip field set as 127.0.0.1 even though X-Forwarded-For header is set in the request.
From X-Ray Concepts:

If a load balancer or other intermediary forwards a request to your application, X-Ray takes the client IP from the X-Forwarded-For header in the request instead of from the source IP in the IP packet.

I wonder if it's due to the use of getLocalIPAddress() method here

    private void populateHeaderToTransactionState(HttpNetworkProtocolRequestEvent requestEvent, XRayTransactionState transactionState) {
        transactionState.withHost(requestEvent.getHost())
                .withMethod(requestEvent.getMethod())
                .withUrl(requestEvent.getURL())
                .withUserAgent(requestEvent.getUserAgent())
                .withClientIP(requestEvent.getLocalIPAddress())
                .withTraceheaderString(requestEvent.getHeaderData(HEADER_KEY));

Hello folks, it looks like the UDP event emitter doesn't not limit the UDP packet size causing these annoying errors:

java.io.IOException: Message too long
at java.base/java.net.PlainDatagramSocketImpl.send(Native Method)
at java.base/java.net.DatagramSocket.send(DatagramSocket.java:695)
at com.amazonaws.xray.emitters.UDPEmitter.sendData(UDPEmitter.java:106)
at com.amazonaws.xray.emitters.UDPEmitter.sendSegment(UDPEmitter.java:84)
at com.amazonaws.xray.AWSXRayRecorder.sendSegment(AWSXRayRecorder.java:178)
at com.amazonaws.xray.contexts.ThreadLocalSegmentContext.endSubsegment(ThreadLocalSegmentContext.java:77)
at com.amazonaws.xray.AWSXRayRecorder.endSubsegment(AWSXRayRecorder.java:546)
at com.amazonaws.xray.AWSXRay.endSubsegment(AWSXRay.java:111)
at com.amazonaws.xray.sql.mysql.TracingInterceptor$TracingStatementProxy.invoke(TracingInterceptor.java:198)
at com.sun.proxy.$Proxy64.execute(Unknown Source)
at jdk.internal.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.tomcat.jdbc.pool.StatementFacade$StatementProxy.invoke(StatementFacade.java:114)
at com.sun.proxy.$Proxy64.execute(Unknown Source)
at org.jdbi.v3.core.statement.SqlLoggerUtil.wrap(SqlLoggerUtil.java:31)
at org.jdbi.v3.core.statement.SqlStatement.internalExecute(SqlStatement.java:1666)
at org.jdbi.v3.core.result.ResultProducers.lambda$returningUpdateCount$0(ResultProducers.java:49)
at org.jdbi.v3.core.statement.Update.execute(Update.java:53)
at org.jdbi.v3.core.statement.Update.execute(Update.java:41)
at com.scanii.core.models.daos.DistributedLockDao.lambda$acquire$0(DistributedLockDao.java:55)
at org.jdbi.v3.core.transaction.LocalTransactionHandler.inTransaction(LocalTransactionHandler.java:134)
at org.jdbi.v3.core.transaction.LocalTransactionHandler.inTransaction(LocalTransactionHandler.java:158)
at org.jdbi.v3.core.Handle.inTransaction(Handle.java:467)
[...]

According to wikipedia (tm), the datagram packet size limit is 64kb (https://en.wikipedia.org/wiki/User_Datagram_Protocol) and yes, there's no good reason for the trace to be that large, but, in any case, it probably makes sense for xray to discard the trace instead of trying so send the packet and throw an exception.

PR #28 adds a workaround to allow Travis CI builds to be executed fully. This workaround imports Disco locally in the project so that Travis CI can compile the agent code without requiring Disco to be published into Maven Central.

Once Disco has been published to Maven Central, these lines should be removed, as well as the disco-lib folders contained in the following subprojects:

• aws-xray-auto-instrumentation-agent-bootstrap
• aws-xray-auto-instrumentation-agent-installer

I recently added aws-xray-java-agent 2.7.1 to an app running Spring boot 2.3.5.

Before I added the agent, startup typically took around 3-4 seconds.
Started Application in 3.927 seconds (JVM running for 4.282)

Using aws-xray-java-agent 2.7.1, the startup of my application takes anywhere between 5-10 times longer.

Basically put the disco jars in place and added this line to my gradle.build

bootRun {
    jvmArgs = ["-javaagent:disco/disco-java-agent.jar=pluginPath=disco/disco-plugins"]
}

Started Application in 21.268 seconds (JVM running for 23.917)

Does this slowdown seem reasonably, or is something going on with my setup?

The XRayListener has a catchall that logs the error of the event. We should make it print out the stack trace as well in order for us to get more informative logging information rather than just the event which is associated with the error

https://github.com/aws/aws-xray-java-agent/blob/master/aws-xray-auto-instrumentation-agent-runtime/src/main/java/com/amazonaws/xray/agent/listeners/XRayListener.java#L50

i have added the agent in classpath and also plugins. but we are unablke to see SQL Traces. Please let me know if there is any configuration file that we need to enable like xray-agent.json to view the SQL traces and examine them. We are able to see http requests but not SQL Queries.

X-ray is getting initialized by below

CATALINA_OPTS='-Xms512M -Xmx1G -javaagent:/path to disco/disco-java-agent.jar=pluginPath=/path to disco/disco-plugins'

Due to the differences in classloading used by Spring Boot vs. other Java apps, the java agent does not respect the configured Logging level for spring boot apps. We should investigate why this is and fix it.

Hello
Sorry to open the issue again

The method that javaagent is started when the app is started executes the premain service logic, but the attach method does not work normally because it enters other service logic while the application is running
Supporting the attach method makes the programming method more beautiful and free
Please review positively. Thank

I have a trivial sample Spring Boot application that I'm attempting to run as an executable jar. The app runs fine without the agent with Jetty and Tomcat. The app also runs fine with the agent when started via Maven. However, starting the executable jar with the documented instructions for including an agent results in the error java.lang.NoClassDefFoundError: javax/servlet/Filter.

This class is definitely available:

[ssm-user@ip-10-0-146-244 merlin]$ unzip -l target/merlin-svc.war | grep javax
    95806  03-17-2023 19:36   WEB-INF/lib/javax.servlet-api-3.1.0.jar
    26586  03-17-2023 19:36   WEB-INF/lib/javax.annotation-api-1.3.2.jar

Working invocations:

./mvnw spring-boot:run -Dspring-boot.run.jvmArguments="-javaagent:/opt/aws/disco/disco-java-agent.jar=pluginPath=/opt/aws/disco/disco-plugins/"

# This only works with the Netty branch
java -jar merlin-svc.war

Failing invocation:

# Same behavior with and without verbose logging, see attached log below
java -javaagent:/opt/aws/disco/disco-java-agent.jar=pluginPath=/opt/aws/disco/disco-plugins/:loggerfactory=software.amazon.disco.agent.reflect.logging.StandardOutputLoggerFactory:verbose -jar /opt/xxxx/merlin/merlin-svc.war

disco.log

Currently there exists a race condition between the Java Agent and Spring when the agent is being bootstrap loaded. This is a known issue and being actively worked on so we can support auto-instrumentation of Spring applications as soon as possible.

When adding the AWS X-Ray Java Agent to a Lambda function via a layer it causes cold starts to be much (30+ seconds) slower. This increase is so dramatic that it is a blocker for having official Lambda support. This is a result of the initialization work the agent must do to auto-instrument your Lambda code, and appears to be a problem with all auto-instrumentation java agents on the Lambda platform. An investigation is needed with the Lambda team to resolve this.

We are trying to instrument our microservices using x-ray java agent. Our microservices are implemented using Spring Boot framework. For other microservices, we accomplished the auto instrumentation using the disco plugins by adding environment variables to our applications as below.

-Dcom.amazonaws.xray.strategy.tracingName=serviceName
-Dcom.amazonaws.xray.strategy.contextMissingStrategy=IGNORE_ERROR
-Dcom.amazonaws.xray.emitter.daemonAddress=127.0.0.1:2000
-Dcom.amazonaws.xray.tracingEnabled=True
-javaagent:src/main/resources/disco/disco-java-agent.jar=pluginPath=src/main/resources/disco/disco-plugins

But in our API Gateway, we failed to do so. While we try to trace any request from our API Gateway, we came across an error like this below:

{
"error": "Exception while serializing entity.",
"stack_trace":"com.fasterxml.jackson.databind.JsonMappingException: Failed to load class
'com.amazonaws.xray.entities.SegmentImpl$Access4JacksonSerializerdb8acfce':
com.fasterxml.jackson.module.afterburner.ser.BeanPropertyAccessor
com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:295)
}
full-stack-trace.json.zip

Also in our x-ray daemon running locally, we encountered with this warning below:

[Warn] Missing header or segment: {"format": "json", "version": 1}

To resolve this error, we tried to exclude plugin path from command like below:

-Dcom.amazonaws.xray.strategy.tracingName=bid-service-local
-Dcom.amazonaws.xray.strategy.contextMissingStrategy=IGNORE_ERROR
-Dcom.amazonaws.xray.emitter.daemonAddress=127.0.0.1:2000
-Dcom.amazonaws.xray.tracingEnabled=True
-javaagent:src/main/resources/disco/disco-java-agent.jar

But there was no segment sent to the daemon that is running locally. Then we tried to add the plugins to its path one by one,

aws-xray-agent-plugin.jar
disco-java-agent-aws-plugin.jar

First, we added the two jars above to disco-plugins path above, there was no error, but segment also not sent to our daemon .

Then, we added the jar below and, we saw that this error is related to this plugin, with some dependencies that we use in our API Gateway application.

disco-java-agent-web-plugin.jar

Dependencies & Technologies

Spring Boot version is 2.3.8.

 - implementation "org.springframework.boot:spring-boot-configuration-processor"
 - implementation "org.springframework.boot:spring-boot-starter-actuator"
 - implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
 - implementation 'org.springframework.cloud:spring-cloud-starter-netflix-zuul'
 - implementation 'org.springframework.cloud:spring-cloud-starter-netflix-hystrix-dashboard'
 - implementation 'org.springframework.cloud:spring-cloud-starter-netflix-hystrix'
 - implementation 'org.springframework.cloud:spring-cloud-starter-netflix-ribbon'
 - implementation "org.springframework.cloud:spring-cloud-starter-sleuth:2.2.8.RELEASE"

Also we are using ZuulFilter to add custom headers to incoming requests' header. Could this also be an issue for our error that we encounter, how can we move forward from here? 🙏

I'm using disco for spring.
Is it possible to specify the dynamic name of disco using YML profile properties or Java config?

Example
Default Name : XRayInstrumentedService
Dynamic Name : Production??

I couldn't find it. But it seemed like there was, so I asked a question

Thank you for checking.

Currently continuous-monitoring workflow uses wget to get the hard coded version of java agent from Maven Central. We should probably add a smoke test for distribution channel and set up a binary metric based on that distribution channel smoke test.

Enabling Trace ID injection via the agent is not currently possible when using Spring Boot. The agent relies on the aws-xray-recorder-sdk-slf4j dependency to be available on the classpath when the agent starts up so it can add the trace ID injector. When the Spring Boot app is packaged with mvn clean package, Spring Boot does packages dependencies weirdly in a way that's not visible to the system class loader when the Agent is starting up.

I think the solution here is to lazy load the trace ID injector, after the spring boot app has started up, once the dependency is discoverable.