Hi,
I've got a problem I'm seeing and I wondered if anyone here could shed any light on it, please?
I was working with GoCD 19.8.0 until last week, then I upgraded to 20.4.0, 20.5.0 and now 20.8.0. I'm using v7.0.0.4 of the plugin "com.thoughtworks.gocd.elastic-agent.ecs", it states it's target version is 19.3.0 but it's the latest version so hopefully compatible with the latest gocd version.
Before I upgraded the gocd server version last week (and started using a postgres db in the process), the gocd-ecs-elastic-agent plugin was working pretty perfectly.
After the upgrade, it's doing a couple of strange things.
I'm seeing errors like, when trying to get an AWS C5d.4xlarge EC2 Spot Request going through the ECS elastic agent plugin (redacted):
jvm 1 | 2020-11-18 22:58:21,052 INFO [154@MessageListener for CreateAgentListener] p.c.t.g.e.e.c.t.g.e.e.ECSElasticPlugin:92 [plugin-com.thoughtworks.gocd.elastic-agent.ecs] - [create-agent] For Platform: 'LINUX',All-Registered-Spot-Instances count: '0', Spot-Requests-Without-Registered-Instances count: '0',UnTagged-Spot-Requests count: '0' jvm 1 | 2020-11-18 22:58:21,214 ERROR [154@MessageListener for CreateAgentListener] p.c.t.g.e.e.c.t.g.e.e.ECSElasticPlugin:127 [plugin-com.thoughtworks.gocd.elastic-agent.ecs] - [create-agent] Failed to create an agent for profile : {"Image":"<gocd-docker-image>","Command":"","Environment":"","MaxMemory":"20G","ReservedMemory":"20G","CPU":"0","MountDockerSocket":"true","Privileged":"true","TaskRoleArn":"arn:aws:iam::1234567890:role/gocd-agents-ops-role","AMI":"ami-XXXXXXXXX","InstanceType":"c5d.4xlarge","SubnetIds":"subnet-XXXXXXXX","SecurityGroupIds":"sg-XXXXXXXX","IAMInstanceProfile":"gocd-agents-ops-profile","Platform":"linux","BindMount":"[\n {\n \"Name\": \"homego\",\n \"SourcePath\": \"/efs\",\n \"ContainerPath\": \"/home/go\"\n }\n]","RunAsSpotInstance":true,"SpotPrice":"0.48","SpotRequestExpiresAfter":"5"} jvm 1 | java.util.NoSuchElementException: null jvm 1 | at java.base/java.util.HashMap$HashIterator.nextNode(Unknown Source) jvm 1 | at java.base/java.util.HashMap$EntryIterator.next(Unknown Source) jvm 1 | at java.base/java.util.HashMap$EntryIterator.next(Unknown Source) jvm 1 | at java.base/java.util.Collections.min(Unknown Source) jvm 1 | at com.thoughtworks.gocd.elasticagent.ecs.aws.SubnetSelector.findSubnetWithMinimumInstances(SubnetSelector.java:54) jvm 1 | at com.thoughtworks.gocd.elasticagent.ecs.aws.SubnetSelector.selectSubnetWithMinimumEC2Instances(SubnetSelector.java:46) jvm 1 | at com.thoughtworks.gocd.elasticagent.ecs.aws.SpotInstanceHelper.requestSpotInstanceRequest(SpotInstanceHelper.java:77) jvm 1 | at com.thoughtworks.gocd.elasticagent.ecs.aws.SpotInstanceService.create(SpotInstanceService.java:108) jvm 1 | at com.thoughtworks.gocd.elasticagent.ecs.aws.TaskHelper.create(TaskHelper.java:80) jvm 1 | at com.thoughtworks.gocd.elasticagent.ecs.ECSTasks.create(ECSTasks.java:69) jvm 1 | at com.thoughtworks.gocd.elasticagent.ecs.executors.CreateAgentRequestExecutor.execute(CreateAgentRequestExecutor.java:63) jvm 1 | at com.thoughtworks.gocd.elasticagent.ecs.ECSElasticPlugin.handle(ECSElasticPlugin.java:99) jvm 1 | at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:134) jvm 1 | at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:208) jvm 1 | at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:164) jvm 1 | at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:131) jvm 1 | at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49) jvm 1 | at com.thoughtworks.go.plugin.access.elastic.v5.ElasticAgentExtensionV5.createAgent(ElasticAgentExtensionV5.java:141) jvm 1 | at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.createAgent(ElasticAgentExtension.java:59) jvm 1 | at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.createAgent(ElasticAgentPluginRegistry.java:42) jvm 1 | at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:31) jvm 1 | at com.thoughtworks.go.server.messaging.elasticagents.CreateAgentListener.onMessage(CreateAgentListener.java:21) jvm 1 | at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83) jvm 1 | at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63) jvm 1 | at java.base/java.lang.Thread.run(Unknown Source)
I'm using only one subnet in that elastic agent configuration (although I was, until today, using two subnets in the form of
,
so, as it currently stands, using just one subnet, it shouldn't be trying to check for which subnet has the least agents in it as it's been told to only look at one of them?
Also, I'm seeing this new log entry set floating around in the logs which means little to me:
jvm 1 | 2020-11-18 23:02:21,100 INFO [qtp247343347-91286] AgentRegistrationController:270 - [Agent Auto Registration] Auto registering agent with uuid 8a41eea9-cf84-4564-b0b1-c67cf35c60e8 jvm 1 | 2020-11-18 23:02:21,107 INFO [155@MessageListener for CreateAgentListener] p.c.t.g.e.e.c.t.g.e.e.ECSElasticPlugin:72 [plugin-com.thoughtworks.gocd.elastic-agent.ecs] - [create-agent] No running instances found to build container with profile {"Image":"480569139359.dkr.ecr.eu-west-2.amazonaws.com/gocd-agents-ops-service-repository","Command":"","Environment":"","MaxMemory":"20G","ReservedMemory":"20G","CPU":"0","MountDockerSocket":"true","Privileged":"true","TaskRoleArn":"arn:aws:iam::1234567890:role/gocd-agents-ops-role","AMI":"ami-XXXXXXXX","InstanceType":"c5d.4xlarge","SubnetIds":"subnet-XXXXXXXX","SecurityGroupIds":"sg-XXXXXXXX","IAMInstanceProfile":"gocd-agents-ops-profile","Platform":"linux","BindMount":"[\n {\n \"Name\": \"homego\",\n \"SourcePath\": \"/efs\",\n \"ContainerPath\": \"/home/go\"\n }\n]","RunAsSpotInstance":true,"SpotPrice":"0.48","SpotRequestExpiresAfter":"5"} jvm 1 | 2020-11-18 23:02:21,388 WARN [qtp247343347-85020] AgentService:539 - Agent [Agent [ea4f74712f3b, 172.18.0.2, 8a41eea9-cf84-4564-b0b1-c67cf35c60e8, null]] has no cookie set jvm 1 | 2020-11-18 23:02:21,388 ERROR [qtp247343347-85020] BuildRepositoryRemoteImpl:59 - Error occurred in com.thoughtworks.go.server.service.ElasticAgentRuntimeInfo@7e549d6[elasticAgentId=GoCDa6528ce340204c54b8fcac556edbe215,elasticPluginId=com.thoughtworks.gocd.elastic-agent.ecs,identifier=Agent [ea4f74712f3b, 172.18.0.2, 8a41eea9-cf84-4564-b0b1-c67cf35c60e8],runtimeStatus=Idle,buildingInfo=AgentBuildingInfo[, ],location=/go,usableSpace=26689011712,operatingSystemName=Ubuntu 18.04 bionic,agentBootstrapperVersion=UNKNOWN,agentVersion=20.8.0-12213,cookie=<null>] ping. jvm 1 | com.thoughtworks.go.server.service.AgentNoCookieSetException: Agent [Agent [ea4f74712f3b, 172.18.0.2, 8a41eea9-cf84-4564-b0b1-c67cf35c60e8, null]] has no cookie set jvm 1 | at com.thoughtworks.go.server.service.AgentService.bombIfAgentDoesNotHaveCookie(AgentService.java:540) jvm 1 | at com.thoughtworks.go.server.service.AgentService.updateRuntimeInfo(AgentService.java:232) jvm 1 | at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.ping(BuildRepositoryRemoteImpl.java:52) jvm 1 | at com.thoughtworks.go.server.messaging.BuildRepositoryMessageProducer.ping(BuildRepositoryMessageProducer.java:44) jvm 1 | at jdk.internal.reflect.GeneratedMethodAccessor491.invoke(Unknown Source) jvm 1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) jvm 1 | at java.base/java.lang.reflect.Method.invoke(Unknown Source) jvm 1 | at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) jvm 1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) jvm 1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) jvm 1 | at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:78) jvm 1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) jvm 1 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) jvm 1 | at com.sun.proxy.$Proxy118.ping(Unknown Source) jvm 1 | at jdk.internal.reflect.GeneratedMethodAccessor769.invoke(Unknown Source) jvm 1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) jvm 1 | at java.base/java.lang.reflect.Method.invoke(Unknown Source) jvm 1 | at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:212) jvm 1 | at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:39) jvm 1 | at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78) jvm 1 | at org.springframework.remoting.support.RemoteInvocationBasedExporter.invokeAndCreateResult(RemoteInvocationBasedExporter.java:114) jvm 1 | at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.handleRequest(HttpInvokerServiceExporter.java:80) jvm 1 | at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:51) jvm 1 | at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) jvm 1 | at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) jvm 1 | at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) jvm 1 | at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) jvm 1 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) jvm 1 | at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) jvm 1 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) jvm 1 | at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395) jvm 1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617) jvm 1 | at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176) jvm 1 | at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145) jvm 1 | at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92) jvm 1 | at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:381) jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) jvm 1 | at com.thoughtworks.go.server.web.FlashLoadingFilter.doFilterInternal(FlashLoadingFilter.java:38) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.DenyIfRefererIsNotFilesFilter.doFilterInternal(DenyIfRefererIsNotFilesFilter.java:52) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.VerifyAuthorityFilter.doFilterInternal(VerifyAuthorityFilter.java:57) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.AbstractUserEnabledCheckFilter.doFilterInternal(AbstractUserEnabledCheckFilter.java:66) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.ThreadLocalUserFilter.doFilterInternal(ThreadLocalUserFilter.java:41) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.AgentAuthenticationFilter.tokenBasedFilter(AgentAuthenticationFilter.java:103) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.AgentAuthenticationFilter.doFilterInternal(AgentAuthenticationFilter.java:65) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.AlwaysCreateSessionFilter.doFilterInternal(AlwaysCreateSessionFilter.java:40) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.ModeAwareFilter.doFilter(ModeAwareFilter.java:79) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at com.thoughtworks.go.server.web.BackupFilter.doFilterInternal(BackupFilter.java:79) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) jvm 1 | at com.thoughtworks.go.server.newsecurity.filterchains.MainFilterChain.doFilter(MainFilterChain.java:79) jvm 1 | at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) jvm 1 | at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) jvm 1 | at com.thoughtworks.go.server.web.DefaultHeadersFilter.doFilter(DefaultHeadersFilter.java:51) jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596) jvm 1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) jvm 1 | at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767) jvm 1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) jvm 1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) jvm 1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) jvm 1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300) jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) jvm 1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) jvm 1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) jvm 1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215) jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) jvm 1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) jvm 1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) jvm 1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) jvm 1 | at org.eclipse.jetty.server.Server.handle(Server.java:500) jvm 1 | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) jvm 1 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) jvm 1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) jvm 1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) jvm 1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) jvm 1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) jvm 1 | at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) jvm 1 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) jvm 1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) jvm 1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) jvm 1 | at java.base/java.lang.Thread.run(Unknown Source) jvm 1 | 2020-11-18 23:02:21,401 WARN [qtp247343347-85020] RemoteInvocationTraceInterceptor:88 - Processing of HttpInvokerServiceExporter remote call resulted in fatal exception: com.thoughtworks.go.remote.BuildRepositoryRemote.ping jvm 1 | org.springframework.remoting.RemoteAccessException: Agent [Agent [ea4f74712f3b, 172.18.0.2, 8a41eea9-cf84-4564-b0b1-c67cf35c60e8, null]] has no cookie set; nested exception is com.thoughtworks.go.server.service.AgentNoCookieSetException: Agent [Agent [ea4f74712f3b, 172.18.0.2, 8a41eea9-cf84-4564-b0b1-c67cf35c60e8, null]] has no cookie set jvm 1 | at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.wrappedException(BuildRepositoryRemoteImpl.java:119) jvm 1 | at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.ping(BuildRepositoryRemoteImpl.java:60) jvm 1 | at com.thoughtworks.go.server.messaging.BuildRepositoryMessageProducer.ping(BuildRepositoryMessageProducer.java:44) jvm 1 | at jdk.internal.reflect.GeneratedMethodAccessor491.invoke(Unknown Source) jvm 1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) jvm 1 | at java.base/java.lang.reflect.Method.invoke(Unknown Source) jvm 1 | at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) jvm 1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) jvm 1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) jvm 1 | at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:78) jvm 1 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) jvm 1 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) jvm 1 | at com.sun.proxy.$Proxy118.ping(Unknown Source) jvm 1 | at jdk.internal.reflect.GeneratedMethodAccessor769.invoke(Unknown Source) jvm 1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) jvm 1 | at java.base/java.lang.reflect.Method.invoke(Unknown Source) jvm 1 | at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:212) jvm 1 | at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:39) jvm 1 | at org.springframework.remoting.support.RemoteInvocationBasedExporter.invoke(RemoteInvocationBasedExporter.java:78) jvm 1 | at org.springframework.remoting.support.RemoteInvocationBasedExporter.invokeAndCreateResult(RemoteInvocationBasedExporter.java:114) jvm 1 | at org.springframework.remoting.httpinvoker.HttpInvokerServiceExporter.handleRequest(HttpInvokerServiceExporter.java:80) jvm 1 | at org.springframework.web.servlet.mvc.HttpRequestHandlerAdapter.handle(HttpRequestHandlerAdapter.java:51) jvm 1 | at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) jvm 1 | at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) jvm 1 | at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) jvm 1 | at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) jvm 1 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) jvm 1 | at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) jvm 1 | at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) jvm 1 | at org.eclipse.jetty.servlet.ServletHolder$NotAsyncServlet.service(ServletHolder.java:1395) jvm 1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:755) jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617) jvm 1 | at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176) jvm 1 | at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145) jvm 1 | at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92) jvm 1 | at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:381) jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) jvm 1 | at com.thoughtworks.go.server.web.FlashLoadingFilter.doFilterInternal(FlashLoadingFilter.java:38) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.DenyIfRefererIsNotFilesFilter.doFilterInternal(DenyIfRefererIsNotFilesFilter.java:52) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.VerifyAuthorityFilter.doFilterInternal(VerifyAuthorityFilter.java:57) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.AbstractUserEnabledCheckFilter.doFilterInternal(AbstractUserEnabledCheckFilter.java:66) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.ThreadLocalUserFilter.doFilterInternal(ThreadLocalUserFilter.java:41) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.AgentAuthenticationFilter.tokenBasedFilter(AgentAuthenticationFilter.java:103) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.AgentAuthenticationFilter.doFilterInternal(AgentAuthenticationFilter.java:65) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.AlwaysCreateSessionFilter.doFilterInternal(AlwaysCreateSessionFilter.java:40) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at com.thoughtworks.go.server.newsecurity.filters.ModeAwareFilter.doFilter(ModeAwareFilter.java:79) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at com.thoughtworks.go.server.web.BackupFilter.doFilterInternal(BackupFilter.java:79) jvm 1 | at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) jvm 1 | at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) jvm 1 | at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) jvm 1 | at com.thoughtworks.go.server.newsecurity.filterchains.MainFilterChain.doFilter(MainFilterChain.java:79) jvm 1 | at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347) jvm 1 | at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263) jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) jvm 1 | at com.thoughtworks.go.server.web.DefaultHeadersFilter.doFilter(DefaultHeadersFilter.java:51) jvm 1 | at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596) jvm 1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) jvm 1 | at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:767) jvm 1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590) jvm 1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) jvm 1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610) jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) jvm 1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1300) jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) jvm 1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) jvm 1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580) jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) jvm 1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1215) jvm 1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) jvm 1 | at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221) jvm 1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) jvm 1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) jvm 1 | at org.eclipse.jetty.server.Server.handle(Server.java:500) jvm 1 | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) jvm 1 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) jvm 1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) jvm 1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) jvm 1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) jvm 1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) jvm 1 | at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) jvm 1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) jvm 1 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) jvm 1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) jvm 1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) jvm 1 | at java.base/java.lang.Thread.run(Unknown Source) jvm 1 | Caused by: com.thoughtworks.go.server.service.AgentNoCookieSetException: Agent [Agent [ea4f74712f3b, 172.18.0.2, 8a41eea9-cf84-4564-b0b1-c67cf35c60e8, null]] has no cookie set jvm 1 | at com.thoughtworks.go.server.service.AgentService.bombIfAgentDoesNotHaveCookie(AgentService.java:540) jvm 1 | at com.thoughtworks.go.server.service.AgentService.updateRuntimeInfo(AgentService.java:232) jvm 1 | at com.thoughtworks.go.remote.BuildRepositoryRemoteImpl.ping(BuildRepositoryRemoteImpl.java:52) jvm 1 | ... 131 common frames omitted
I can't see anything useful on the web about that error, just a gocd-gitter comment or two after a few hours of googling, and even less about the first error.
Is anyone else seeing these problems? If anyone could shed any light on what is causing these, if it's a misconfiguration onh my part etc, it'd be greatly appreciated.
Thanks!
Steve