I've been following the instructions to get Red Dog up and running on my ASA instance. The order-service initially failed. I had to add SERVICEBUSCONNECTIONSTRING to the az spring app create command.
But now loyalty-service and makeline-service app deployments are failing and I am not unable to understand why. Below is the log output for the loyalty-service.
mike@LAPTOP-E5O4743B:/mnt/c/projects/reddog-code-spring$ az spring app logs -n $SERVICE_NAME -s $SPRING_CLUSTER -g $RG --lines 1000
Build in Environment Variables
BUILD_IN_EUREKA_CLIENT_SERVICEURL_DEFAULTZONE=https://asareddog27153mike11974.svc.azuremicroservices.io/eureka/eureka
BUILD_IN_SPRING_CLOUD_CONFIG_URI=https://asareddog27153mike11974.svc.azuremicroservices.io/config
BUILD_IN_SPRING_CLOUD_CONFIG_FAILFAST=true
[Azure Spring Cloud] The following environment variables are loaded: KAFKA_COMPLETED_ORDERS_TOPIC KAFKA_CONSUMER_GROUP_ID AZUREREDISHOST AZUREREDISACCESSKEY REQUIRED AZURESTORAGEACCOUNTNAME KAFKASASLMECHANISM AZURECOSMOSDBKEY MYSQLURL AZURECOSMOSDBDATABASENAME AZURECOSMOSDBURI MYSQLPASSWORD PASSWORD KAFKASECURITYPROTOCOL KAFKATOPICNAME AZURESTORAGEENDPOINT USERNAME KAFKASASLJAASCONFIG AZUREREDISPORT KAFKABOOTSTRAPSERVERS AZURESTORAGEACCOUNTKEY SERVICEBUSCONNECTIONSTRING KAFKATOPICGROUP MYSQLUSER
Setting Active Processor Count to 16
Adding $JAVA_OPTS to $JAVA_TOOL_OPTIONS
Calculated JVM Memory Configuration: -XX:MaxDirectMemorySize=10M -Xmx392953K -XX:MaxMetaspaceSize=143622K -XX:ReservedCodeCacheSize=240M -Xss1M (Total Memory: 1G, Thread Count: 250, Loaded Class Count: 22943, Headroom: 0%)
Enabling Java Native Memory Tracking
Adding 127 container CA certificates to JVM truststore
Spring Cloud Bindings Enabled
Picked up JAVA_TOOL_OPTIONS: -Djava.security.properties=/layers/paketo-buildpacks_microsoft-openjdk/java-security-properties/java-security.properties -XX:+ExitOnOutOfMemoryError -XX:ActiveProcessorCount=16 -XX:InitialRAMPercentage=60.0 -XX:MaxRAMPercentage=60.0 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.local.only=true -Dmanagement.endpoints.jmx.exposure.include=health,metrics -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dspring.jmx.enabled=true -Dserver.tomcat.mbeanregistry.enabled=true -Dfile.encoding=UTF8 -Deureka.client.service-url.defaultZone=https://asareddog27153mike11974.svc.azuremicroservices.io/eureka/eureka -DCONFIG_SERVER_URL=https://asareddog27153mike11974.svc.azuremicroservices.io/config -Dspring.cloud.config.uri=https://asareddog27153mike11974.svc.azuremicroservices.io/config -Dspring.cloud.config.fail-fast=true -XX:MaxDirectMemorySize=10M -Xmx392953K -XX:MaxMetaspaceSize=143622K -XX:ReservedCodeCacheSize=240M -Xss1M -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -Dorg.springframework.cloud.bindings.boot.enable=true
__ __ _ _ __ __ ____ _ _
( ) / \ ( \/ ) / _\ ( ) (_ _)( \/ )
/ (_/\( O ) ) / / \/ (_/\ )( ) /
\____/ \__/ (__/ \_/\_/\____/ (__) (__/
____ ____ ____ _ _ __ ___ ____
/ ___)( __)( _ \/ )( \( ) / __)( __)
\___ \ ) _) ) /\ \/ / )( ( (__ ) _)
(____/(____)(__\_) \__/ (__) \___)(____)
2023-02-03 02:45:21.744 INFO [loyalty-service,,] 1 --- [ main] c.m.g.r.l.LoyaltyServiceApplication : Starting LoyaltyServiceApplication v0.0.1-SNAPSHOT using Java 17.0.4.1 on loyalty-service-default-15-868cdcc9c-mk6z4 with PID 1 (/workspace/BOOT-INF/classes started by cnb in /workspace)
2023-02-03 02:45:21.818 INFO [loyalty-service,,] 1 --- [ main] c.m.g.r.l.LoyaltyServiceApplication : No active profile set, falling back to 1 default profile: "default"
2023-02-03 02:45:21.937 WARN [loyalty-service,,] 1 --- [ main] o.s.b.c.config.ConfigDataEnvironment : Property 'spring.profiles' imported from location 'class path resource [application.yaml]' is invalid and should be replaced with 'spring.config.activate.on-profile' [origin: class path resource [application.yaml] - 56:13]
2023-02-03 02:45:24.840 INFO [loyalty-service,,] 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Multiple Spring Data modules found, entering strict repository configuration mode
2023-02-03 02:45:24.843 INFO [loyalty-service,,] 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data Redis repositories in DEFAULT mode.
2023-02-03 02:45:24.921 INFO [loyalty-service,,] 1 --- [ main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 66 ms. Found 0 Redis repository interfaces.
2023-02-03 02:45:25.464 INFO [loyalty-service,,] 1 --- [ main] o.s.cloud.context.scope.GenericScope : BeanFactory id=cdb1659e-e66b-3399-8209-0da12c40d8f3
2023-02-03 02:45:27.024 INFO [loyalty-service,,] 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 1025 (http)
2023-02-03 02:45:27.353 INFO [loyalty-service,,] 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2023-02-03 02:45:27.353 INFO [loyalty-service,,] 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.65]
2023-02-03 02:45:27.574 INFO [loyalty-service,,] 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2023-02-03 02:45:27.574 INFO [loyalty-service,,] 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 5635 ms
2023-02-03 02:45:35.327 INFO [loyalty-service,,] 1 --- [ main] DiscoveryClientOptionalArgsConfiguration : Eureka HTTP Client uses RestTemplate.
2023-02-03 02:45:35.946 WARN [loyalty-service,,] 1 --- [ main] iguration$LoadBalancerCaffeineWarnLogger : Spring Cloud LoadBalancer is currently working with the default cache. While this cache implementation is useful for development and tests, it's recommended to use Caffeine cache in production.You can switch to using Caffeine cache, by adding it and org.springframework.cache.caffeine.CaffeineCacheManager to the classpath.
2023-02-03 02:45:36.141 INFO [loyalty-service,,] 1 --- [ main] o.s.b.a.e.web.EndpointLinksResolver : Exposing 16 endpoint(s) beneath base path '/actuator'
2023-02-03 02:45:36.543 INFO [loyalty-service,,] 1 --- [ main] o.a.k.clients.consumer.ConsumerConfig : ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [ehreddog27153mike11974.servicebus.windows.net:9093]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = consumer-loyalty-service-1
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = loyalty-service
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor, class org.apache.kafka.clients.consumer.CooperativeStickyAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = [hidden]
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.connect.timeout.ms = null
sasl.login.read.timeout.ms = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.login.retry.backoff.max.ms = 10000
sasl.login.retry.backoff.ms = 100
sasl.mechanism = PLAIN
sasl.oauthbearer.clock.skew.seconds = 30
sasl.oauthbearer.expected.audience = null
sasl.oauthbearer.expected.issuer = null
sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
sasl.oauthbearer.jwks.endpoint.url = null
sasl.oauthbearer.scope.claim.name = scope
sasl.oauthbearer.sub.claim.name = sub
sasl.oauthbearer.token.endpoint.url = null
security.protocol = SASL_SSL
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 45000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
value.deserializer = class org.springframework.kafka.support.serializer.JsonDeserializer
2023-02-03 02:45:36.649 INFO [loyalty-service,,] 1 --- [ main] org.apache.kafka.common.metrics.Metrics : Metrics scheduler closed
2023-02-03 02:45:36.649 INFO [loyalty-service,,] 1 --- [ main] org.apache.kafka.common.metrics.Metrics : Closing reporter org.apache.kafka.common.metrics.JmxReporter
2023-02-03 02:45:36.650 INFO [loyalty-service,,] 1 --- [ main] org.apache.kafka.common.metrics.Metrics : Metrics reporters closed
2023-02-03 02:45:36.651 INFO [loyalty-service,,] 1 --- [ main] o.a.kafka.common.utils.AppInfoParser : App info kafka.consumer for consumer-loyalty-service-1 unregistered
2023-02-03 02:45:36.651 WARN [loyalty-service,,] 1 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
2023-02-03 02:45:36.728 INFO [loyalty-service,,] 1 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2023-02-03 02:45:36.753 INFO [loyalty-service,,] 1 --- [ main] ConditionEvaluationReportLoggingListener :
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2023-02-03 02:45:36.828 ERROR [loyalty-service,,] 1 --- [ main] o.s.boot.SpringApplication : Application run failed
org.springframework.context.ApplicationContextException: Failed to start bean 'org.springframework.kafka.config.internalKafkaListenerEndpointRegistry'; nested exception is org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:181) ~[spring-context-5.3.22.jar:5.3.22]
at org.springframework.context.support.DefaultLifecycleProcessor.access$200(DefaultLifecycleProcessor.java:54) ~[spring-context-5.3.22.jar:5.3.22]
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.start(DefaultLifecycleProcessor.java:356) ~[spring-context-5.3.22.jar:5.3.22]
at java.base/java.lang.Iterable.forEach(Iterable.java:75) ~[na:na]
at org.springframework.context.support.DefaultLifecycleProcessor.startBeans(DefaultLifecycleProcessor.java:155) ~[spring-context-5.3.22.jar:5.3.22]
at org.springframework.context.support.DefaultLifecycleProcessor.onRefresh(DefaultLifecycleProcessor.java:123) ~[spring-context-5.3.22.jar:5.3.22]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:935) ~[spring-context-5.3.22.jar:5.3.22]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586) ~[spring-context-5.3.22.jar:5.3.22]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:147) ~[spring-boot-2.7.2-SNAPSHOT.jar:2.7.2-SNAPSHOT]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:734) ~[spring-boot-2.7.2-SNAPSHOT.jar:2.7.2-SNAPSHOT]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:408) ~[spring-boot-2.7.2-SNAPSHOT.jar:2.7.2-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:308) ~[spring-boot-2.7.2-SNAPSHOT.jar:2.7.2-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306) ~[spring-boot-2.7.2-SNAPSHOT.jar:2.7.2-SNAPSHOT]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295) ~[spring-boot-2.7.2-SNAPSHOT.jar:2.7.2-SNAPSHOT]
at com.microsoft.gbb.reddog.loyaltyservice.LoyaltyServiceApplication.main(LoyaltyServiceApplication.java:12) ~[classes/:0.0.1-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[workspace/:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[workspace/:na]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[workspace/:na]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) ~[workspace/:na]
Caused by: org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:823) ~[kafka-clients-3.1.1.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:664) ~[kafka-clients-3.1.1.jar:na]
at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createRawConsumer(DefaultKafkaConsumerFactory.java:483) ~[spring-kafka-2.8.8.jar:2.8.8]
at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createKafkaConsumer(DefaultKafkaConsumerFactory.java:451) ~[spring-kafka-2.8.8.jar:2.8.8]
at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createConsumerWithAdjustedProperties(DefaultKafkaConsumerFactory.java:427) ~[spring-kafka-2.8.8.jar:2.8.8]
at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createKafkaConsumer(DefaultKafkaConsumerFactory.java:394) ~[spring-kafka-2.8.8.jar:2.8.8]
at org.springframework.kafka.core.DefaultKafkaConsumerFactory.createConsumer(DefaultKafkaConsumerFactory.java:371) ~[spring-kafka-2.8.8.jar:2.8.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.<init>(KafkaMessageListenerContainer.java:776) ~[spring-kafka-2.8.8.jar:2.8.8]
at org.springframework.kafka.listener.KafkaMessageListenerContainer.doStart(KafkaMessageListenerContainer.java:352) ~[spring-kafka-2.8.8.jar:2.8.8]
at org.springframework.kafka.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:461) ~[spring-kafka-2.8.8.jar:2.8.8]
at org.springframework.kafka.listener.ConcurrentMessageListenerContainer.doStart(ConcurrentMessageListenerContainer.java:209) ~[spring-kafka-2.8.8.jar:2.8.8]
at org.springframework.kafka.listener.AbstractMessageListenerContainer.start(AbstractMessageListenerContainer.java:461) ~[spring-kafka-2.8.8.jar:2.8.8]
at org.springframework.kafka.config.KafkaListenerEndpointRegistry.startIfNecessary(KafkaListenerEndpointRegistry.java:347) ~[spring-kafka-2.8.8.jar:2.8.8]
at org.springframework.kafka.config.KafkaListenerEndpointRegistry.start(KafkaListenerEndpointRegistry.java:292) ~[spring-kafka-2.8.8.jar:2.8.8]
at org.springframework.context.support.DefaultLifecycleProcessor.doStart(DefaultLifecycleProcessor.java:178) ~[spring-context-5.3.22.jar:5.3.22]
... 22 common frames omitted
Caused by: java.lang.IllegalArgumentException: Login module control flag not specified in JAAS config
at org.apache.kafka.common.security.JaasConfig.parseAppConfigurationEntry(JaasConfig.java:110) ~[kafka-clients-3.1.1.jar:na]
at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:63) ~[kafka-clients-3.1.1.jar:na]
at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:88) ~[kafka-clients-3.1.1.jar:na]
at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:82) ~[kafka-clients-3.1.1.jar:na]
at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:167) ~[kafka-clients-3.1.1.jar:na]
at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81) ~[kafka-clients-3.1.1.jar:na]
at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105) ~[kafka-clients-3.1.1.jar:na]
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:736) ~[kafka-clients-3.1.1.jar:na]
... 36 common frames omitted
Native Memory Tracking:
Total: reserved=954047886, committed=587173262
- Java Heap (reserved=402653184, committed=402653184)
(mmap: reserved=402653184, committed=402653184)
- Class (reserved=122897548, committed=9520268)
(classes #13885)
( instance classes #12951, array classes #934)
(malloc=1262732 #30510)
(mmap: reserved=121634816, committed=8257536)
( Metadata: )
( reserved=58720256, committed=55115776)
( used=54833368)
( waste=282408 =0.51%)
( Class space:)
( reserved=121634816, committed=8257536)
( used=8029288)
( waste=228248 =2.76%)
- Thread (reserved=21108240, committed=1857040)
(thread #20)
(stack: reserved=21049344, committed=1798144)
(malloc=37632 #130)
(arena=21264 #38)
- Code (reserved=255330160, committed=24823664)
(malloc=1701744 #7901)
(mmap: reserved=253628416, committed=23121920)
- GC (reserved=1343170, committed=1343170)
(malloc=24258 #83)
(mmap: reserved=1318912, committed=1318912)
- Compiler (reserved=237224, committed=237224)
(malloc=66696 #756)
(arena=170528 #7)
- JVMCI (reserved=104, committed=104)
(malloc=104 #7)
- Internal (reserved=435246, committed=435246)
(malloc=398382 #9524)
(mmap: reserved=36864, committed=36864)
- Other (reserved=16384, committed=16384)
(malloc=16384 #2)
- Symbol (reserved=15308984, committed=15308984)
(malloc=13860320 #364046)
(arena=1448664 #1)
- Native Memory Tracking (reserved=6710808, committed=6710808)
(malloc=5752 #81)
(tracking overhead=6705056)
- Shared class space (reserved=12582912, committed=12447744)
(mmap: reserved=12582912, committed=12447744)
- Arena Chunk (reserved=55873480, committed=55873480)
(malloc=55873480)
- Tracing (reserved=33041, committed=33041)
(malloc=313 #6)
(arena=32728 #1)
- Logging (reserved=5220, committed=5220)
(malloc=5220 #219)
- Arguments (reserved=3325, committed=3325)
(malloc=3325 #113)
- Module (reserved=406832, committed=406832)
(malloc=406832 #2657)
- Safepoint (reserved=8192, committed=8192)
(mmap: reserved=8192, committed=8192)
- Synchronization (reserved=77184, committed=77184)
(malloc=77184 #864)
- Serviceability (reserved=984, committed=984)
(malloc=984 #14)
- Metaspace (reserved=58990000, committed=55385520)
(malloc=269744 #185)
(mmap: reserved=58720256, committed=55115776)
- String Deduplication (reserved=704, committed=704)
(malloc=704 #8)
- Object Monitors (reserved=24960, committed=24960)
(malloc=24960 #120)
![mend-bolt-for-github[bot] avatar](https://avatars.githubusercontent.com/in/16809?v=4 "mend-bolt-for-github[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent