When can support redis

Write a simple documentation for parts where some explanation is needed. For example RestTemplates are traced only when they are injected as beans (and it's a default way how to access it in spring), however some might ask why manually created RestTemplates are not traced.

At the beginning, we could just use Readme in opentracing-spring-cloud directory.

It would be great to be able to exclude some @scheduled job.

Our specific usecase is that we run some health check on a scheduled and we would like to ignore that.

I'll try to think of how it could be implemented and report back

I'm not entirely sure where the issue is. But here is the behavior I'm observing when using the opentracing-spring-cloud-feign tracing module.

1. The Spring Boot application that consumes this module goes from a ~15 sec startup time to ~5 min startup time.
2. From log files it seems to indicate excessive RMI activity to be the culprit
3. Renaming the tracer variable in FeignTracingAutoConfiguration.java to anything but tracer resolves the issue

I have good and bad trace level startup logs I can share for comparison.

The readme currently shows how to configure, but not how a springboot app could obtain and use the tracer. Might be good to have a simple code snippet.

Because Hystrix internally uses RxJava it introduce some problems.
If RxJava instrumentation is enabled (via auto configuration) then Hystrix (and feign-hystrix) traces include RxJava spans.
All hystrix-feign tests fails because instead of one finished spans there tens of finished spans.

2 spaces, 4 spaces instead of tabs etc...

• The proxy uses Ribbon to locate an instance to forward to via discovery, and all requests are executed in a hystrix command
• The default HTTP client used by zuul is now backed by the Apache HTTP Client instead of the deprecated Ribbon RestClient. To use RestClient or to use the okhttp3.OkHttpClient set ribbon.restclient.enabled=true or ribbon.okhttp.enabled=true respectively.

A very valid point has been raised that the auto-configuration modules depend on Spring Cloud making them heavy for simple REST apps that don't need all the Spring Cloud baggage.

We could move the jaeger and zipkin implementation into a separate repo and have them depend on either java-spring-web or java-spring-cloud, much like the old snowdrop implementation did.

Can I have your thoughts @pavolloffay @objectiser @gytis @cmoulliard ?

There are two possible integrations:

• put SpanContext into standard logs - there is a problem that OpenTracing does not define getters on SpanContext therefore it's not possible to extract any IDs from it.
• redirect standard logging to OT logs

Edgware is stable, but README doesn't says if it is supported.

Hi,
.editorconfig files should be included in the project, since it is using non-standard indentation for Java files. It is annoying to prepare pull requests, and many PRs will have incorrect indentation because this file is missing. But I've found some comments regarding this in #42.

Why it was refused? It is intended exactly for this purpose - to provide same formatting rules for everyone, regardless of their IDE and local settings (most modern IDEs support it) and should be part of the sources.

It can be as simple as this (although a lot can be tuned to match required preference):

root = true

[*]
indent_size = 2
continuation_indent_size = 4

Create auto-configuration for https://github.com/OpenFeign/feign-opentracing.

Auto configuration should instrument all interfaces annotated with @FeignClient but also clients created manually. HystrixFeign should be also supported.

Some links:

• docs http://cloud.spring.io/spring-cloud-netflix/spring-cloud-netflix.html#spring-cloud-feign

Test this library with Spring Boot Actuator https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#production-ready.

We should probably disable tracing for actuator endpoints. And maybe do other adjustments if needed.

I was wondering if it was ever discuss to have a @Traced method-level annotation?

It would allow to easily start/stop a span around a method (using AOP). We could even capture the args and return value at some point.

Something like this:

@Target({ ElementType.METHOD, ElementType.ANNOTATION_TYPE })
@Retention(RetentionPolicy.RUNTIME)
public @interface Traced {

    boolean withActiveSpanOnly() default true;

    String component() default "java-traced-annotation";
    
    String operationName() default "";

}

Apache SkyWalking offers this: https://github.com/apache/incubator-skywalking/blob/master/apm-application-toolkit/apm-toolkit-trace/src/main/java/org/apache/skywalking/apm/toolkit/trace/Trace.java

And it could be similar to the @Timed annotation from DropWizard http://metrics.dropwizard.io/3.1.0/apidocs/com/codahale/metrics/annotation/Timed.html

https://docs.spring.io/spring-cloud-stream/docs/current/reference/htmlsingle/

Support to propagate the spans across threads when using spring where javanica @HystrixCommand annotations are used

opentracing-spring-cloud-starter instruments all artifacts what are on classpath. It might be useful to have more control and disable instrumentation with property e.g. opentracing.spring.cloud.feign=false

I have a spring-boot/spring-cloud application using opentracing (See https://github.com/Hawkular-QE/jaeger-instrumentation-performance-tests/tree/master/spring-boot) which produces the stack trace below when run with any version of opentracing-spring-cloud-starter newer than 0.0.2.

I can get the following combinations of dependencies to work correctly:

• spring-boot 1.5.6.RELEASE, spring-cloud Dalston.SR3, opentracing-spring-cloud-starter 0.0.2
• spring-boot 1.5.8.RELEASE, spring-cloud Dalston.SR4, opentracing-spring-cloud-starter 0.0.2

However, either combination with the opentracing version changed to 0.0.4, 0.0.5, or 0.0.6-SNAPSHOT produces the ClassNotFoundException shown below.

To reproduce, clone [email protected]:Hawkular-QE/jaeger-instrumentation-performance-tests.git and then execute: mvn -f spring-boot/pom.xml clean install spring-boot:run

Caused by: java.lang.NoClassDefFoundError: org/springframework/messaging/support/ChannelInterceptor
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getDeclaredConstructors(Class.java:2020)
at org.springframework.cglib.proxy.Enhancer.generateClass(Enhancer.java:566)
at org.springframework.cglib.transform.TransformingClassGenerator.generateClass(TransformingClassGenerator.java:33)
at org.springframework.cglib.core.DefaultGeneratorStrategy.generate(DefaultGeneratorStrategy.java:25)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanFactoryAwareGeneratorStrategy.generate(ConfigurationClassEnhancer.java:252)
at org.springframework.cglib.core.AbstractClassGenerator.generate(AbstractClassGenerator.java:329)
at org.springframework.cglib.proxy.Enhancer.generate(Enhancer.java:492)
at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:93)
at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:91)
at org.springframework.cglib.core.internal.LoadingCache$2.call(LoadingCache.java:54)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.springframework.cglib.core.internal.LoadingCache.createEntry(LoadingCache.java:61)
... 26 more
Caused by: java.lang.ClassNotFoundException: org.springframework.messaging.support.ChannelInterceptor
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 40 more

Pivotal just released the following security vulnerability:

CVE-2018-1270: Remote Code Execution with spring-messaging

Severity: Critical
Vendor: Spring by Pivotal

Description

Spring Framework versions 5.0 to 5.0.4, 4.3 to 4.3.14, and older unsupported versions allow applications to expose STOMP over WebSocket endpoints with a simple, in-memory STOMP broker through the spring-messaging module. A malicious user (or attacker) can craft a message to the broker that can lead to a remote code execution attack.
Affected Pivotal Products and Versions

Severity is critical unless otherwise noted.

Spring Framework 5.0 to 5.0.4
Spring Framework 4.3 to 4.3.14
Older unsupported versions are also affected

Mitigation

Users of affected versions should apply the following mitigation:

5.0.x users should upgrade to 5.0.5
4.3.x users should upgrade to 4.3.15

There are no other mitigation steps necessary.

The use of authentication and authorization of messages, such as the one provided by Spring Security, can limit exposure to this vulnerability only to users who are allowed to use the application.
Credit

This issue was identified and responsibly reported by Alvaro Muñoz (@pwntester), Micro Focus Fortify.
References

Example STOMP over WebSocket config where simple broker enabled.
Spring Security WebSocket Support.

History

2018-04-05: Initial vulnerability report published

Create auto-configuration which would trace RxJava. This https://github.com/opentracing-contrib/java-rxjava instrumentation should be reused.

Spring Cloud has several release trains:

• Camden.*
• Brixton.*
• Dalston.*
  etc.

README doesn't say anything about supported train.
Is it safe to add instrumentation to any release?
It will not introduce jar hell if I will add it to existing spring cloud app of any version?

I expect that all versions should come from

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-dependencies</artifactId>
    <version>???</version>
    <type>pom</type>
    <scope>import</scope>
</dependency>

Original issue: jaegertracing/jaeger-client-java#381

Hi,

I have been using jaeger-core 0.24.0 version along with opentracing-spring-cloud-starter 0.1.1 version.
One of the library in our project uses spring-cloud-context version 1.3.2.RELEASE.
After using spring cloud context jar, we are not getting traces properly.
I am getting in downstream service span.

cc @chandresh-pancholi

Links:

• https://github.com/gavlyukovskiy/spring-boot-data-source-decorator
• https://github.com/opentracing-contrib/java-jdbc

https://docs.spring.io/spring/docs/current/spring-framework-reference/html/scheduling.html#scheduling-annotation-support-scheduled

might be related to #35

Not sure if it's important, but because Zuul is instrumented probably Gateway needs instrumentation or maybe works because of spring web instrumentation...
Need to check

It looks like support is missing for:

• org.springframework.data.redis.connection.RedisConnectionFactory
• org.springframework.data.mongodb.MongoDbFactory

Support for these could be leveraged on top of java-redis-client and java-mongo-driver, respectively (both in opentracing-contrib).

Trace org.springframework.messaging.MessageChannel

When instrumenting a service using websockets, it also reports spans for the HTTP communications to establish the inbound and outbound connections: #53 (comment)

Options:

• Leave as is, and use this information to know when and how many connections are being made. If so - might be good to see if websocket message spans can be correlated to the session connection spans in some way
• Automatically register skip patterns for the inbound/outbound path prefixes

#97 introduced redirect of standard logging into span logs.

Logback supports filters https://logback.qos.ch/manual/filters.html to change a decision on appending log. Right now we send all logs to span (log of the level of root logger). However, this might overload tracing system. Just note that some tracing systems might strip long log strings.

Question if how to support this. Autowire list of Filter<E>? This might autowire other filters from the context, a solution might be to use classifier like tracingLogFilter.

cc @jfbreault

• web tracing + skip pattern
• scheduled + skip
• messaging
• logging
• how to configure vendor starters - jaeger

Currently WebsocketAutoConfiguration is enabled by default, and it does not respect whether websocket are included in the application at all.
In addition to opentracing.spring.cloud.websocket.enabled property, it should be also conditional on class WebSocketMessageBrokerConfigurationSupport, which is part of spring-websocket maven artifact.

There is already @ConditionalOnBean(WebSocketMessageBrokerConfigurationSupport.class), but that is not enough, it needs to be @ConditionalOnClass as well.

Workaround is to set

opentracing.spring.cloud.websocket.enabled=false

Note: I'm trying this with Spring Boot 2, but same problem should be with 1.5.x.

Application error (stacktrace cleaned up a little):

2018-03-01 14:51:31.531 ERROR 27121 --- [  restartedMain] o.s.boot.SpringApplication               : Application run failed
java.lang.IllegalStateException: Cannot load configuration class: io.opentracing.contrib.spring.cloud.websocket.WebsocketAutoConfiguration
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.enhanceConfigurationClasses(ConfigurationClassPostProcessor.java:416) ~[spring-context-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanFactory(ConfigurationClassPostProcessor.java:254) ~[spring-context-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:284) ~[spring-context-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:128) ~[spring-context-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:693) ~[spring-context-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:531) ~[spring-context-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:138) ~[spring-boot-2.0.0.RC1.jar:2.0.0.RC1]
	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:752) [spring-boot-2.0.0.RC1.jar:2.0.0.RC1]
	at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:388) [spring-boot-2.0.0.RC1.jar:2.0.0.RC1]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:327) [spring-boot-2.0.0.RC1.jar:2.0.0.RC1]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1246) [spring-boot-2.0.0.RC1.jar:2.0.0.RC1]
	at org.springframework.boot.SpringApplication.run(SpringApplication.java:1234) [spring-boot-2.0.0.RC1.jar:2.0.0.RC1]
Caused by: java.lang.IllegalStateException: Unable to load cache item
Caused by: java.lang.NoClassDefFoundError: org/springframework/web/socket/config/annotation/WebSocketMessageBrokerConfigurationSupport
	at org.springframework.cglib.core.ReflectUtils.addAllMethods(ReflectUtils.java:416) ~[spring-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.cglib.proxy.Enhancer.getMethods(Enhancer.java:540) ~[spring-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.cglib.proxy.Enhancer.generateClass(Enhancer.java:575) ~[spring-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.cglib.transform.TransformingClassGenerator.generateClass(TransformingClassGenerator.java:33) ~[spring-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.cglib.core.DefaultGeneratorStrategy.generate(DefaultGeneratorStrategy.java:25) ~[spring-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanFactoryAwareGeneratorStrategy.generate(ConfigurationClassEnhancer.java:253) ~[spring-context-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.cglib.core.AbstractClassGenerator.generate(AbstractClassGenerator.java:329) ~[spring-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.cglib.proxy.Enhancer.generate(Enhancer.java:492) ~[spring-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:93) ~[spring-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.cglib.core.AbstractClassGenerator$ClassLoaderData$3.apply(AbstractClassGenerator.java:91) ~[spring-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at org.springframework.cglib.core.internal.LoadingCache$2.call(LoadingCache.java:54) ~[spring-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
	at org.springframework.cglib.core.internal.LoadingCache.createEntry(LoadingCache.java:61) ~[spring-core-5.0.3.RELEASE.jar:5.0.3.RELEASE]
	... 25 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.springframework.web.socket.config.annotation.WebSocketMessageBrokerConfigurationSupport
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:582) ~[na:na]
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:185) ~[na:na]
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:496) ~[na:na]
	... 41 common frames omitted

spring-data is popular data-access framework. This task should verify that our current JDBC instrumentation works with https://docs.spring.io/spring-data/jpa/docs/1.11.6.RELEASE/reference/html/.

related to #27, #38

The README file mentions to use arbitrary OpenTracing tracer within Sleuth section.
Can you clarify and give an example about what is an arbitrary OpenTracing tracer ?

Add make target to test this Finchley with SB 2.0.0

I would like to discuss moving/implementing Spring auto-configuration for different vendors into contrib org. The motivation is to make these starters more visible and easier to use. For example for Jaeger there is already a starter [1], but it's very hard to find it.

There are several ways how this can be done:

1. each starter in a separate repo
2. one repo for all starters
3. include starters directly in this repo

[1] https://github.com/snowdrop/opentracing-tracer

cc @tedsuo @malafeev @yurishkuro @objectiser

Not necessarily an issue, but just want to clarify how best to make a Tracer available to various spring and non-spring based instrumentations.

To set the context, the OT java API provides a utility called GlobalTracer to allow independent instrumentation code to access a singleton tracer. In spring based apps, the preferred approach is to be able to inject a Tracer.

Where auto-config is used to configure either a spring or non-spring based framework instrumentation, it is possible that the injection approach could be used. However there may be a problem if the framework instrumentation code itself uses the GlobalTracer.

Therefore for the spring-cloud auto-config work, do we assume that an inject only approach will be used - and where instrumentation is found that currently uses GlobalTracer, we provide PRs to enable a non-GlobalTracer approach to be used?

Or as part fo the auto-config work, do we ensure that the same tracer used for injection is also registered with the GlobalTracer, to make sure we are independent of whichever approach is used by the framework instrumentations?

When tests are executed from command line there are several exceptions.
It causes that one test executed from IDE passes but from command line fails.

1. Actuator test

https://travis-ci.org/opentracing-contrib/java-spring-cloud/jobs/287093279

Test in actuator branch

2. JdbcTracingTest always fails for me from command line, but passes from IDE.

RestTemplate instances annotated with @LoadBalanced are traced, however ping requests made by Ribbon are not.

#2 (comment)

cc @geoand

Ideally there should be a way, possibly as part of the starter, to initialize the Tracer based on a set of defined steps.

For example,

1. If the application has provided their own Tracer bean, then it should take precedence over any other option, and be registered with the GlobalTracer (for use by other non-spring instrumentations)
2. If no application provided bean is available, and the TracerResolver is available, then it should be used. Once again - if this returns a value then it should be registered with the GlobalTracer.

The only potential conflict that could occur is if before this initialization has been performed, the application has explicitly registered a Tracer with the GlobalTracer that is different from one available via the spring context. If this occurs, then an error should be logged, as it will potentially result in tracing information being reported to multiple tracers.

Instrumentation artifact should not pull any dependencies if users do not explicitly specify it.

We are currently using the optional scope for any 3rd party deps except instrumentation artifacts. Instrumentation artifacts (e.g. feign-hystrix-opentracing) are compile scope and exclude their dependencies (e.g. feign-core).

We should consider adding tests to address

• by default artifact should not pull any 3rd party deps
• artifact should not fails to start due to class not found or simillar errors.

cd opentracing-spring-cloud 
mvn clean install 

fails on

[ERROR] Failed to execute goal com.mycila:license-maven-plugin:3.0:check (default) on project opentracing-spring-cloud: Execution default of goal com.mycila:license-maven-plugin:3.0:check failed: Cannot read header document header.txt. Cause: Resource header.txt not found in file system, classpath or URL: no protocol: header.txt -> [Help 1]

• trace org.springframework.scheduling.annotation.Async
• wrap org.springframework.scheduling.annotation.AsyncConfigurer with traced alternative

#87 (comment)

We might consider adding a starter artifact to align with spring convention.

cc @alesj @cmoulliard