Comments (23)
jairojunior
commented on July 28, 2024
net/http default timeout is 60s, which is a reasonable value, but we'll probably need a new timeout param for wildfly_deploy in order to override default timeout.
I don't think it's necessary for wildfly_resource and wildfly_cli, since all sync operations that I know are really fast (miliseconds).
Thank you for your contributions. I'll give them more attention as soon I can.
from puppet-wildfly.
jairojunior
commented on July 28, 2024
Just did the change in PR #73
Can you test it? I only have EARs that took < 50s.
By the way, be careful about these hibernate checks if you're puppetizing your entire infrastructure. Internode dependencies are somehow problematic, if your deploy fails cause of the database (hibernate.hbm2ddl.auto = validate), it means you'll always have to provision your database before your Application Server.
I suggest you watch this presentation: https://puppetlabs.com/presentations/puppetizing-multitier-architecture-reid-vandewiele-puppet-labs
It's extremely insightful. =)
from puppet-wildfly.
smbambling
commented on July 28, 2024
I pulled this down and I'm still seeing the timeouts. Do you want me to make a PR to update the ::wildfly::deploy manifest to add the timeout param that can be set.
Noted on your presentation link I'll def take a look at that. Unfortunately currently some stuff gets 'tossed over the wall' from our DEV department and we just have to work with it.
from puppet-wildfly.
jairojunior
commented on July 28, 2024
You're right. We need to update ::wildfly::deploy. Go ahead and submit your PR if you make it work, just wanted to give you a little incentive.
Yours DEV department probably have a point, hibernate.hbm2ddl.auto = validate is a nice feature, but they could be easily doing this validation in an integration test.
Puppet is a paradigm shift in app development since it seeks for eventual consistency, not traditional all or nothing model we are used to.
from puppet-wildfly.
smbambling
commented on July 28, 2024
I'm seeing some odd inconstant behavior with the timeouts. With the PR that you merged in I modified the type to default to 300 for testing purposes. When that changes is made I get an error from running puppet apply that causes the deployment of the WAR to be rolled back
Error: Could not set 'present' on ensure: Net::ReadTimeout at 61:/vagrant/modules/wildfly/manifests/deploy.pp
Error: Could not set 'present' on ensure: Net::ReadTimeout at 61:/vagrant/modules/wildfly/manifests/deploy.pp
Wrapped exception:
Net::ReadTimeout
Error: /Stage[main]/Whoisrws::Deploy/Wildfly::Deploy[whoisrws_proxy.war]/Wildfly_deploy[whoisrws_proxy.war]/ensure: change from absent to present failed: Could not set 'present' on ensure: Net::ReadTimeout at 61:/vagrant/modules/wildfly/manifests/deploy.pp
In the server.log I see this error
2015-09-02 08:37:04,397 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 65) MSC000001: Failed to start service jboss.deployment.unit."whoisrws_proxy.war".component.WhoisProxyMBeanImpl.START: org.jboss.msc.service.StartException in service jboss.deployment.unit."whoisrws_proxy.war".component.WhoisProxyMBeanImpl.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:163)
at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:122)
at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:136)
at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
... 6 more
Caused by: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleEndTransactionException(CMTTxInterceptor.java:137)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:117)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349)
at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:66)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
... 11 more
Caused by: javax.transaction.RollbackException: WFLYEJB0447: Transaction 'TransactionImple < ac, BasicAction: 0:ffff7f000001:-31679b84:55e6ebb5:1a status: ActionStatus.ABORTED >' was already rolled back
at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:98)
... 28 more
If I leave the timeout to 120 as set in your PR, I get a Net::ReadTimeout and the WAR DOES deploy. I also get this same behavior if I set the timeout to 60 sec as per the default for Net::ReadTimeout
Error: Could not set 'present' on ensure: Net::ReadTimeout at 61:/vagrant/modules/wildfly/manifests/deploy.pp
Error: Could not set 'present' on ensure: Net::ReadTimeout at 61:/vagrant/modules/wildfly/manifests/deploy.pp
Wrapped exception:
Net::ReadTimeout
Error: /Stage[main]/Whoisrws::Deploy/Wildfly::Deploy[whoisrws_proxy.war]/Wildfly_deploy[whoisrws_proxy.war]/ensure: change from absent to present failed: Could not set 'present' on ensure: Net::ReadTimeout at 61:/vagrant/modules/wildfly/manifests/deploy.pp
from puppet-wildfly.
jairojunior
commented on July 28, 2024
I'll deeply investigate it tonight. Just build a JAR that takes 120s to be deployed:
@Singleton
@Startup
public class StartupService {
@PostConstruct
public void init() throws InterruptedException {
System.out.println("Waiting...");
Thread.sleep(120000);
System.out.println("Done");
}
}
from puppet-wildfly.
jairojunior
commented on July 28, 2024
Just tested locally and made it work with the above code in a deployable JAR and timeout = 150. Can you confirm its working so we can push a new version?
from puppet-wildfly.
smbambling
commented on July 28, 2024
Sure can you bump your timeout to 300 and let me know what behavior you see
from puppet-wildfly.
jairojunior
commented on July 28, 2024
Sure. I'm still confused about timeout behavior when interacting with JBoss HTTP API and not sure if we should be handling this flow...
What I observed is that even when the connection timeouts, the operation is still processed. (i.e. HTTP connection timeout doesn't abort the operation.)
from puppet-wildfly.
smbambling
commented on July 28, 2024
I also saw that behavior with lower timeout values...but if the the timeout was set to a higher value like 300 then it would roll back the deployment.
I'm also retesting and documenting my iterations.
from puppet-wildfly.
smbambling
commented on July 28, 2024
I was just able to make the deployment fail by only upping the timeout value to 240s. At that point I get an error in the server.log
2015-09-04 10:48:06,902 ERROR [org.jboss.as.controller.management-operation] (XNIO-2 task-6) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'deploy' at address '[("deployment" => "whoisrws_proxy.war")]'
2015-09-04 10:48:06,904 ERROR [org.jboss.as.server] (XNIO-2 task-6) WFLYSRV0021: Deploy of deployment "whoisrws_proxy.war" was rolled back with the following failure message: "WFLYCTL0344: Operation timed out awaiting service container stability"
2015-09-04 10:48:07,425 WARN [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff7f000001:-53bf0485:55e9ad5a:1b in state RUN
2015-09-04 10:48:07,427 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012095: Abort of action id 0:ffff7f000001:-53bf0485:55e9ad5a:1b invoked while multiple threads active within it.
2015-09-04 10:48:07,427 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012108: CheckedAction::check - atomic action 0:ffff7f000001:-53bf0485:55e9ad5a:1b aborting with 1 threads active!
2015-09-04 10:48:07,427 WARN [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff7f000001:-53bf0485:55e9ad5a:1b
2015-09-04 10:48:07,587 WARN [com.arjuna.ats.arjuna] (ServerService Thread Pool -- 72) ARJUNA012077: Abort called on already aborted atomic action 0:ffff7f000001:-53bf0485:55e9ad5a:1b
2015-09-04 10:48:07,589 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 72) MSC000001: Failed to start service jboss.deployment.unit."whoisrws_proxy.war".component.WhoisProxyMBeanImpl.START: org.jboss.msc.service.StartException in service jboss.deployment.unit."whoisrws_proxy.war".component.WhoisProxyMBeanImpl.START: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:57)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: java.lang.IllegalStateException: WFLYEE0042: Failed to construct component instance
at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:163)
at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:134)
at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:88)
at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:122)
at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:136)
at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54)
... 6 more
Caused by: javax.ejb.EJBTransactionRolledbackException: Transaction rolled back
at org.jboss.as.ejb3.tx.CMTTxInterceptor.handleEndTransactionException(CMTTxInterceptor.java:137)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:117)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInOurTx(CMTTxInterceptor.java:279)
at org.jboss.as.ejb3.tx.CMTTxInterceptor.requiresNew(CMTTxInterceptor.java:349)
at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:66)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.weld.injection.WeldInjectionContextInterceptor.processInvocation(WeldInjectionContextInterceptor.java:43)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.as.ee.concurrent.ConcurrentContextInterceptor.processInvocation(ConcurrentContextInterceptor.java:45)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.ContextClassLoaderInterceptor.processInvocation(ContextClassLoaderInterceptor.java:64)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.InterceptorContext.run(InterceptorContext.java:356)
at org.jboss.invocation.PrivilegedWithCombinerInterceptor.processInvocation(PrivilegedWithCombinerInterceptor.java:80)
at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:340)
at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61)
at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161)
... 11 more
Caused by: javax.transaction.RollbackException: WFLYEJB0447: Transaction 'TransactionImple < ac, BasicAction: 0:ffff7f000001:-53bf0485:55e9ad5a:1b status: ActionStatus.ABORTED >' was already rolled back
at org.jboss.as.ejb3.tx.CMTTxInterceptor.endTransaction(CMTTxInterceptor.java:98)
... 28 more
from puppet-wildfly.
smbambling
commented on July 28, 2024
I've gotten some really mixed results with when deploys will fail and be rolled back. Maybe like you hinted to above in place of having the HTTP call to timeout you use the timeout parameter to wait X seconds and then exec an additional CLI check to see if the WAR was successful deployed
/deployment=jolokia-war-agent.war:read-attribute(name=status)
This puts the onus on the developer of the manifest that calls the wildly::cli::deploy define to make sure they have the correct timeout and ordering in place.
from puppet-wildfly.
smbambling
commented on July 28, 2024
@jairojunior do you have an thoughts on my above comment of splitting up the process and making having a specific call to check the status of an deployed app ?
from puppet-wildfly.
jairojunior
commented on July 28, 2024
If you think about Puppet eventual consistency mode, we shouldn't be looking for this sort of guarantee.
I believe that both JBoss and HTTP client are able to issue a timeout and that is probably the cause of the rollback (i.e. if JBoss timeouts he is able to revert an operation), but I need to investigate with tcpdump to confirm.
from puppet-wildfly.
smbambling
commented on July 28, 2024
I think your correct that the default JBoss timeout seems to be 300sec which causes the rollback.
I still don't understand the issue with the HTTP client reporting back a failure to set present even though the WAR still deploys ( the WAR is still in the process of deploying after the timeout is triggered ) when the timeout is set lower then 300 sec.
What is the 'Puppet eventual consistency mode' your referring to ?
Why wouldn't you want to look for a guarantee that the app was successfully deployed and reported as so via the application server (Jboss) ?
from puppet-wildfly.
jairojunior
commented on July 28, 2024
If you run puppet three times it reports that application is already deployed (i.e. eventual consistency). Right?
I don't think that everyone should "pay the price" (extra verification) for this edge case. I'll investigate HTTP API to understand what is really happening and check what kind of verification is possible.
It's just my opinion based on my experience with JBoss HTTP API because still didn't have time to dig into this problem. =/
from puppet-wildfly.
jairojunior
commented on July 28, 2024
I want to bring this thread up again after a lot of discussion with my colleagues regarding this subject.
I'm inclined to believe that some operations (JBoss-wise) shouldn't be performed by Puppet, but by some sort of Orchestrator (like Collective, Fabric and Capistrano)
Based on this thought I created a new project called mcollective-jboss-agent (https://github.com/jairojunior/mcollective-jboss-agent) and you'll find more information can be found in the README.md of this project.
@biemond you have an opinion about this? By the way, we are ready to deliver 0.4.2
from puppet-wildfly.
smbambling
commented on July 28, 2024
This sounds like a great project but doesn't/shouldn't stop the need for having puppet manage wars.
Initial WAR deployment via has the benefits of verifying that state of your system is consistent. While using an orchestration tool seems like a better fit for updating your deployments.
Outside of the current issue where a war is being deployed but that state of the deployment is not being verifyed its very easy to make a profile that will deploy wars and still be idempotent.
from puppet-wildfly.
jairojunior
commented on July 28, 2024
You're right. We still need resources (wildfly_deploy) to declare and manage our infrastructure state.
from puppet-wildfly.
jairojunior
commented on July 28, 2024
Maybe we need to rename wildfly_deploy to wildfly_deployment since it's how JBoss calls it:
<deployment name="MyApp.ear" runtime-name="MyApp.ear">
<content sha1="e067140a5f0696faaad7b981b36b5b34033a1f19"/>
</deployment>
from puppet-wildfly.
mpeter
commented on July 28, 2024
try this:
<property name="jboss.as.management.blocking.timeout" value="1800"/>
from puppet-wildfly.
smbambling
commented on July 28, 2024
Is anyone at PuppetConf this week, would be cool to talk face to face.
from puppet-wildfly.
jairojunior
commented on July 28, 2024
@mpeter Interesting property.
@smbambling Unfortunately not. Maybe next year... It's a long way from Brazil to Portland. =/
from puppet-wildfly.
Related Issues (20)
- Error: Could not set 'file' on ensure: No such file or directory @ dir_s_mkdir HOT 2
- JBOSS-EAP with **systemctl start jboss-eap** don't works from another machine just locally HOT 1
- where the log file is stored on Puppet Agent by this module HOT 1
- on SUSE Linux /etc/init.d/jboss-eap: line 13: /etc/init.d/functions: No such file or directory HOT 1
- wildfly_restart functionality issues
- how to install und build multiple standalone instances in the same machine
- Need a way to install module with multiple jar files as source HOT 1
- Transfer this module to voxpupuli HOT 22
- wildfly::jgroups::stack::tcpping fails with "No resource definition" HOT 1
- Update test environment for Ruby 3, Puppet 7, and related Gems HOT 1
- Add parameters to domain.conf variables PROCESS_CONTROLLER_JAVA_OPTS and HOST_CONTROLLER_JAVA_OPTS
- Replace MaxPermSize with MaxMetaspaceSize in domain.conf template HOT 1
- `wildfly::host::server_config` with `ensure => absent` in already absent server tries to stop it and fails with timeout
- Include support for CLI API for actions currently performed via HTTP API HOT 2
- Multiple calls to defined type wildfly::host::server_config cause "Duplicate resource" error HOT 3
- class wildfly::setup hardcodes subsystem when setting up remote user HOT 2
- Installer download fails because install_cache_dir doesn't exist
- Module configures service only as sysvinit, regardless of actual init system, when $version < 10 HOT 1
- Update Wildfly download URL for version >= 25
- Support for Wildfly 31.0.0
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from puppet-wildfly.