spring-attic / core Goto Github PK
View Code? Open in Web Editor NEWCore components shared by other projects in the app starters organization
License: Apache License 2.0
Core components shared by other projects in the app starters organization
License: Apache License 2.0
I think we should update org.springframework.cloud:spring-cloud-build
version to 1.2.1.RELEASE
Move https://github.com/spring-cloud-stream-app-starters/load-generator/blob/master/spring-cloud-starter-stream-source-load-generator/src/main/java/org/springframework/cloud/stream/app/load/generator/source/ContentTypeEnvironmentPostProcessor.java so it can be shared across other app projects needing the same functionality
As a developer, I'd like to add "micrometer-spring-legacy
and app-starters-micrometer-common
dependencies to 1.3.x branch.
Acceptance:
app-starters-micrometer-common
module to 1.3.x@tzolov commented on Fri Mar 20 2020
The prometheus metrics for Fahrenheit apps is broken
[ERROR] testMetadataStore[4](org.springframework.cloud.stream.app.metadata.MetadataStoreAutoConfigurationTests) Time elapsed: 2.718 s <<< ERROR!
java.lang.IllegalStateException: Unstarted application context org.springframework.boot.test.context.assertj.AssertableApplicationContext[startupFailure=org.springframework.context.ApplicationContextException] failed to start
at org.springframework.cloud.stream.app.metadata.MetadataStoreAutoConfigurationTests.lambda$testMetadataStore$1(MetadataStoreAutoConfigurationTests.java:106)
at org.springframework.cloud.stream.app.metadata.MetadataStoreAutoConfigurationTests.testMetadataStore(MetadataStoreAutoConfigurationTests.java:105)
Caused by: org.springframework.context.ApplicationContextException: Failed to start bean 'zookeeperMetadataStore'; nested exception is java.lang.NoSuchMethodError: org.apache.curator.framework.api.ExistsBuilder.creatingParentContainersIfNeeded()Lorg/apache/curator/framework/api/ACLableExistBuilderMain;
at org.springframework.cloud.stream.app.metadata.MetadataStoreAutoConfigurationTests.testMetadataStore(MetadataStoreAutoConfigurationTests.java:105)
Caused by: java.lang.NoSuchMethodError: org.apache.curator.framework.api.ExistsBuilder.creatingParentContainersIfNeeded()Lorg/apache/curator/framework/api/ACLableExistBuilderMain;
at org.springframework.cloud.stream.app.metadata.MetadataStoreAutoConfigurationTests.testMetadataStore(MetadataStoreAutoConfigurationTests.java:105)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] MetadataStoreAutoConfigurationTests.testMetadataStore:105->lambda$testMetadataStore$1:106 » IllegalState
[INFO]
[ERROR] Tests run: 8, Failures: 0, Errors: 1, Skipped: 0
The upgrade to Boot from 2.1.3
to 2.1.6
, Fishtown.SR2
and Greenwich.SR2
is compulsory foor enabling the Prometheus RSocket Proxy micrometer (doesn't work for Boot versions below 2.1.5) but it seems to causes errors on CF environments (below). Looks like problem is with spring-cloud-spring-service-connector-2.0.5.RELEASE.jar
derived from spring-boot-2.1.6.jar
Above changes don't cause issues on Local, minikube and GKE/k8s
This is the log from CF deploying time
source BS:
2019-09-14T13:30:04.449+02:00 [APP/PROC/WEB/0] [OUT] java.lang.NoClassDefFoundError: org/springframework/jdbc/datasource/DelegatingDataSource
2019-09-14T13:30:04.449+02:00 [APP/PROC/WEB/0] [OUT] at java.lang.ClassLoader.defineClass1(Native Method) ~[na:1.8.0_222]
2019-09-14T13:30:04.449+02:00 [APP/PROC/WEB/0] [OUT] at java.lang.ClassLoader.defineClass(ClassLoader.java:763) ~[na:1.8.0_222]
2019-09-14T13:30:04.449+02:00 [APP/PROC/WEB/0] [OUT] at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) ~[na:1.8.0_222]
2019-09-14T13:30:04.449+02:00 [APP/PROC/WEB/0] [OUT] at java.net.URLClassLoader.defineClass(URLClassLoader.java:468) ~[na:1.8.0_222]
2019-09-14T13:30:04.449+02:00 [APP/PROC/WEB/0] [OUT] at java.net.URLClassLoader.access$100(URLClassLoader.java:74) ~[na:1.8.0_222]
2019-09-14T13:30:04.449+02:00 [APP/PROC/WEB/0] [OUT] at java.net.URLClassLoader$1.run(URLClassLoader.java:369) ~[na:1.8.0_222]
2019-09-14T13:30:04.449+02:00 [APP/PROC/WEB/0] [OUT] at java.net.URLClassLoader$1.run(URLClassLoader.java:363) ~[na:1.8.0_222]
2019-09-14T13:30:04.449+02:00 [APP/PROC/WEB/0] [OUT] at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_222]
2019-09-14T13:30:04.449+02:00 [APP/PROC/WEB/0] [OUT] at java.net.URLClassLoader.findClass(URLClassLoader.java:362) ~[na:1.8.0_222]
2019-09-14T13:30:04.449+02:00 [APP/PROC/WEB/0] [OUT] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[na:1.8.0_222]
2019-09-14T13:30:04.449+02:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:92) ~[app/:na]
This has side effects for green field apps
Enable security for all the actuator endpoints.
The markersJson = false
means we don't want to convert FileMarker
to JSON and just want to send it as serialized
, but since we supply "text/plain"
in the FileUtils
:
flowBuilder.enrichHeaders(Collections.<String, Object>singletonMap(MessageHeaders.CONTENT_TYPE,
"text/plain"))
.split(new FileSplitter(true, withMarkers, fileConsumerProperties.getMarkersJson()));
the org.springframework.cloud.stream.converter.ObjectStringMessageConverter
does its trick
and just call Object.toString()
on our FileMarker
instead of expected serialization.
And even if we would serialize it manually, that's going to be confusing on the consumer side since contentType == text/plain
.
Another way might be like conditional header enricher and override the contentType
header to the application/x-java-serialized-object
if payload is FileMarker
.
On the consumer side you can have conditional @StreamListener
to handle string
with one method
and FileMarker
s with another.
From @sabbyanandan on March 26, 2018 22:44
As a developer, I'd like to add actuator and webflux dependencies, so I can enable /bindings
endpoint by default across all the apps.
Here's the relevant section in the docs.
Acceptance:
/bindings
via management.endpoints.web.exposure.include=bindings
property in all the appsCopied from original issue: spring-attic/app-starters-release#136
As a developer, I'd like to upgrade kafka-binder to be compatible with 0.10 release of Apache Kafka as a default.
Use SpEL to all message contents to be passed to tasks as args. For example:
file --spring.cloud.stream.function.definition=taskLaunchRequest --task.launch.request.task-name=foo --task.launch.request.arg-expressions='localFilePath=' + payload
We will be removing CF connector library from the apps.
By default only info
and health
endpoints permit an unauthorized access.
For accessing the /actuator/prometheus
we should either
[ERROR] dataflowTaskLaunchRequestWithTaskLaunchRequestContextProvider(org.springframework.cloud.stream.app.tasklaunchrequest.TaskLaunchRequestIntegrationTests) Time elapsed: 0.156 s <<< ERROR!
org.springframework.beans.factory.support.BeanDefinitionOverrideException: Invalid bean definition with name 'binderFactory' defined in class path resource [org/springframework/cloud/stream/test/binder/TestSupportBinderAutoConfiguration.class]: Cannot register bean definition [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.cloud.stream.test.binder.TestSupportBinderAutoConfiguration; factoryMethodName=binderFactory; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/cloud/stream/test/binder/TestSupportBinderAutoConfiguration.class]] for bean 'binderFactory': There is already [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=bindingServiceConfiguration; factoryMethodName=binderFactory; initMethodName=null; destroyMethodName=(inferred); defined in org.springframework.cloud.stream.config.BindingServiceConfiguration] bound.
at org.springframework.cloud.stream.app.tasklaunchrequest.TaskLaunchRequestIntegrationTests.dataflowTaskLaunchRequestWithTaskLaunchRequestContextProvider(TaskLaunchRequestIntegrationTests.java:114)
[ERROR] dataflowTaskLaunchRequestWithArgsAndDeploymentProperties(org.springframework.cloud.stream.app.tasklaunchrequest.TaskLaunchRequestIntegrationTests) Time elapsed: 0.122 s <<< ERROR!
org.springframework.beans.factory.support.BeanDefinitionOverrideException: Invalid bean definition with name 'binderFactory' defined in class path resource [org/springframework/cloud/stream/test/binder/TestSupportBinderAutoConfiguration.class]: Cannot register bean definition [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=org.springframework.cloud.stream.test.binder.TestSupportBinderAutoConfiguration; factoryMethodName=binderFactory; initMethodName=null; destroyMethodName=(inferred); defined in class path resource [org/springframework/cloud/stream/test/binder/TestSupportBinderAutoConfiguration.class]] for bean 'binderFactory': There is already [Root bean: class [null]; scope=; abstract=false; lazyInit=false; autowireMode=3; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=bindingServiceConfiguration; factoryMethodName=binderFactory; initMethodName=null; destroyMethodName=(inferred); defined in org.springframework.cloud.stream.config.BindingServiceConfiguration] bound.
at org.springframework.cloud.stream.app.tasklaunchrequest.TaskLaunchRequestIntegrationTests.dataflowTaskLaunchRequestWithArgsAndDeploymentProperties(TaskLaunchRequestIntegrationTests.java:74)
[INFO] Running org.springframework.cloud.stream.app.tasklaunchrequest.TaskLaunchRequestPropertiesTests
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.133 s - in org.springframework.cloud.stream.app.tasklaunchrequest.TaskLaunchRequestPropertiesTests
[INFO] Running org.springframework.cloud.stream.app.tasklaunchrequest.DeploymentPropertiesParserTests
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 s - in org.springframework.cloud.stream.app.tasklaunchrequest.DeploymentPropertiesParserTests
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] TaskLaunchRequestIntegrationTests.dataflowTaskLaunchRequestWithArgsAndDeploymentProperties:74 » BeanDefinitionOverride
[ERROR] TaskLaunchRequestIntegrationTests.dataflowTaskLaunchRequestWithTaskLaunchRequestContext:92 » BeanDefinitionOverride
[ERROR] TaskLaunchRequestIntegrationTests.dataflowTaskLaunchRequestWithTaskLaunchRequestContextProvider:114 » BeanDefinitionOverride
[INFO]
[ERROR] Tests run: 11, Failures: 0, Errors: 3, Skipped: 0
As a user, I'd like to monitor KStream apps in SCDF, so when including the stream-apps-micrometer-common
in the CP, I hope to also automatically parse all the various KStreams binding types via this dependency.
This causes to have all the jars for testing in the target generated uber jar for the starter application.
Looks suspicious, but I won't mind if you find some argument and explain at least for me why we need them here.
Thanks
Addressing the following SCDF request: Issue#12933, Spring Boot has implemented a Common Tags feature that allows to declaratively add Common Tags
using just application properties.
Review the current SpringCloudStreamMicrometerCommonTags to see if we can simplify it using the new boot feature.
Currently, depending on the FileReadingMode
mode the FileUtils
sets the following content types:
application/octet-stream
for content
modetext/plain
for lines
modeThe ref
mode though ( https://goo.gl/Xt8K3F ) is left to be set to the default content type! While currently this happens to be application/json
we can not guaranty it will not change in the future. Furthermore this assumption creates an unnecessary coupling between this code (and all apps that use it) and the core SCSt functionality. Instead we should set the content-type explicitly to application/json
similar to the other two modes.
The spring-cloud-stream-metrics
is obsolete in 2.0.0. Furthermore it drags in Spring Boot 1.5.x metrics dependencies casing Class Not Found exceptions.
This is a blocker for the Darwin.M1 release
As a developer, I'd like to review the patching procedure and as well as the customization steps for correctness.
See discussion here for background.
Ship the out-of-the box applications with the following Micrometer dependencies in the classpath
All configured micrometer registries should be disabled by default, e.g.
--management.metrics.export.datadog.enabled=false
--management.metrics.export.influx.enabled=false
--management.metrics.export.prometheus.enabled=false
/actuator/prometheus/
):--management.endpoints.web.exposure.include=prometheus
Also, check for point releases in:
Add the prometheus.rsocket
metrics registry to enable Stream App starters use the Prometheus Rsocket Proxy instead of custom build Service Discovery mechanism.
Really two separate issues but:
Sftp source already provides the ability to optionally transform the output to a Task Launch Request but this is a cross-cutting concern for any message producer. Provide a core/common module to support this feature without requiring a separate transformer.
Sftp source needs to copy remote files to a local directory in most situations. If running in cloudfoundry with the nfs
(Volume Services) service bound, the local directory path should be auto-reconfigured to use the services nfs mount point. This is also a cross-cutting concern for any source that creates files in a local directory. The app configures a @Bean LocalDirectoryResolver
using the default constructor and uses that to resolve the local directory path. The auto reconfiguration provides a BeanPostProcessor to replace this bean if it exists and the above conditions are true.
Currently the task name is a required property, statically configured.
Make kafka apps default to 0.10.1.1
The original idea is in the Tensorflow processor PR: spring-attic/tensorflow#1 (comment).
Since we don't test now spring-configuration-metadata-whitelist.properties
in the app starter we don't know if that is correct for target integration system like SCDF.
With the common security configuration class SecurityCommonAutoConfiguration
's DisableSecurityConfiguration using the default Order
of WebSecurityConfigurerAdapter
which is 100, the other WebSecurityConfigurerAdapter
in the same application configuration conflicts.
For instance, the http
source app doesn't start with this and through the following:
Caused by: java.lang.IllegalStateException: @Order on WebSecurityConfigurers must be unique. Order of 100 was already used on org.springframework.cloud.stream.app.http.source.HttpSourceConfiguration$HttpSourceSecurityConfiguration$$EnhancerBySpringCGLIB$$50356d4e@55562aa9, so it cannot be used on org.springframework.cloud.stream.app.security.common.SecurityCommonAutoConfiguration$DisableSecurityConfiguration$$EnhancerBySpringCGLIB$$57f5ab9a@3bd418e4 too.
As a developer, I'd like to add a new app-starters-metadata-store
module, so we can host auto-configuration code for a variety of MetadataStore
.
Storage options:
In order to migrate testing to use the new SI Test Binder, it is necessary to move spring-cloud-stream-test-support
to core-dependencies
. Since it is present in the parent pom it gets pulled in to all app starters even though it is declared optional
. This breaks the SI Test Binder when used in a Test since it is also test
scope.
Since many app starters implicitly depend on spring-cloud-stream-test-support
, the moved dependency should no longer be declared as optional. This should be restored to optional when the app starters start to migrate to the test binder.
NOTE. In the mean time, app starters that use the SI test binder need to exclude spring-cloud-stream-test-support
<dependency>
<groupId>org.springframework.cloud.stream.app</groupId>
<artifactId>app-starters-core-dependencies</artifactId>
<version>2.0.0.BUILD-SNAPSHOT</version>
<type>pom</type>
<scope>import</scope>
<exclusions>
<exclusion>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-test-support</artifactId>
</exclusion>
</exclusions>
</dependency>
Create an auto-configuration to inject common tags for:
Adding a declarative Micrometer Tags injection utilities, common for all app starters.
The idea is to provide the same fixed/expression based tags enhancement functionality available in the Counter to all app starters.
For example, any app starter should be able to register additional Micrometer tags, with values dynamically resolved from the Message objects using SpEL expressions:
analytics.tag.expression.author=#jsonPath(payload,'$..author')
The PollerMetadata defaultPoller
bean created in org.springframework.cloud.stream.app.trigger.TriggerConfiguration
has an issue around properly setting the max messages per poll. The code is comparing this.triggerProperties.getMaxMessages() > -1
but I believe it should be this.triggerProperties.getMaxMessages() >= -1
. Allowing the max messages per poll to be set to -1
is a necessity for when a polling operation fetches multiple events and queues them up for retrieval allowing them all to be obtained without waiting for the polling period between each message.
As documented in Spring Integration:
Note there is no max-messages-per-poll specified. As you’ll learn later the identical poller configuration in the PollingConsumer (e.g., service-activator, filter, router etc.) would have a default value of -1 for max-messages-per-poll which means "execute poling task non-stop unless polling method returns null (e.g., no more Messages in the QueueChannel)" and then sleep for 1 second.
However in the SourcePollingChannelAdapter it is a bit different. The default value for max-messages-per-poll will be set to 1 by default unless you explicitly set it to a negative value (e.g., -1). It is done so to make sure that poller can react to a LifeCycle events (e.g., start/stop) and prevent it from potentially spinning in the infinite loop if the implementation of the custom method of the MessageSource has a potential to never return null and happened to be non-interruptible.
However if you are sure that your method can return null and you need the behavior where you want to poll for as many sources as available per each poll, then you should explicitly set max-messages-per-poll to a negative value.
The "app-starters-micrometer-common" for 2.0.0.RC1 has changed significantly, since the initial 1.3.x port. Resulting in new tag name convention (dot based), removed tags (e.g. clusterName
) and support for CF tags.
This is a follow up of #38
The time
is a reserved keyword for InfuxDB tag values. As a result the the time
app starters fails to register its name as a metrics application.name
tag:
2019-01-12 13:09:56.410 ERROR 25728 --- [trics-publisher] i.micrometer.influx.InfluxMeterRegistry : failed to send metrics to influx
java.lang.IllegalArgumentException: 'time' is an invalid tag value in InfluxDB
at io.micrometer.influx.InfluxNamingConvention.tagValue(InfluxNamingConvention.java:66) ~[micrometer-registry-influx-1.1.1.jar!/:1.1.1]
at io.micrometer.core.instrument.Meter$Id.lambda$getConventionTags$0(Meter.java:330) ~[micrometer-core-1.1.1.jar!/:1.1.1]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_152]
Quick work around is to rename the time app starter name to time2
for example.
Better solution though is customize the meter registry so it renames the time
tag value to atime
for InfluxDB.
Provide support for any component to output a TaskLaunchRequest via function composition.
As a developer, I'd like to extract add org/space from env-var and add it to cmmonTags
Map, so we could use it to plot metrics associated with streams in CF's Org/Space combination.
Acceptance:
This needs to be done only for CF and that can be determined based on Spring Profile being cloud
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.