puppetlabs / puppetserver Goto Github PK
View Code? Open in Web Editor NEWServer automation framework and application
Home Page: https://tickets.puppetlabs.com/browse/SERVER
License: Apache License 2.0
puppetserver's People
Contributors
Stargazers
Watchers
Forkers
dalen cprice404 wolfspyre thorwang mwbutcher big-sha mslilah dankreek waynr camlow325 terrycorley mcanevet mahak unixbhaskar ahpook kbarber jeffmccune syathish butangero inhumantsar rlinehan nfagerlund madalinignisca elyscape masterzen nicklewis jpinsonault chundu lpalgarvio geoffwilliams cswaroop pdxmph mbrukman arunagummalla cloud-flare petems stahnma peculiarism ehtobsemaj cgvarela nrcxcia zipkid scottyw kylog sengkoil pravinbhagade scotje mullr underscorgan achinthagunasekara mfournier cyberious fiddyspence mmckinst andersonmills heathseals melissa justinstoller fr34k8 jzkeiser malysoun ldenman karenvdv ollie314 janelu2 cadelaren sarameisburger er0ck nmunagala lethaldoze codedeploybyjilu ccin2p3 mikaelsmith digideskio jvperrin lucywyman m4ce tvpartytonight heliocentric magisus mwaggett sharpie aperiodic senior rodriguezjf ipcrm ajroetker jsane iristyle damionjunk mosesmendoza grossjo libbymo thallgren juxtin ricardoamaro ebahmer jtappa ekinanp funk1337puppetserver's Issues
puppetserver 7.x's logged message about hiera() deprecation includes a broken URL
Describe the Bug
puppetserver 7.16 logs a deprecation warning with a broken URL:
2024-03-11T17:43:29.958-05:00 WARN [qtp1601237113-43] [puppetserver] Puppet The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.29/deprecated_language.html (file & line not available)
Expected Behavior
It's very handy that the puppetserver produces a log message with a "go here to learn more" URL. The only thing that needs to change is for the URL it logs to be updated for the current location.
Steps to Reproduce
Steps to reproduce the behavior:
- Use hiera() instead of lookup() somewhere in your puppet code
- restart puppetserver on your puppet server.
- Do a 'no-op' puppet agent run
- browse the /var/log/puppetlabs/puppsetserver/puppetserver.log
- copy the URL and paste it into a browser
- puppet.com's custom 404 page comes up
Environment
- Version: puppetserver 7.16.0, puppet-agent 7.29.1
- Platform: RHEL 9.x
Puppetserver README is out of date
- README.md and github's sidebar contain references to pre-migration JIRA
- MAINTAINERS is gone, replaced by CODEOWNERS
Describe the Change You Would Like
Better pointers to how to file an o/s-related issue against OSP.
Please make the packaging/build pipelines public and provide reproducible builds
Use Case
As a service delivery partner we want to engage into the development. This isn't really possible without at least read-only access to the pipelines. We don't know which jobs exist and what they do. That makes it impossible to add support for new platforms. And as we can see on the existing requests Puppet itself is quite slow for adding support for new distributions/architectures.
Describe the Solution You Would Like
use GitHub actions for supported platforms to build packages (preferred) or at least grant partners access to the Jenkins (which was public in the past).
Describe Alternatives You've Considered
While I hate it that an open source tool uses private pipelines, you could also workaround this by providing new packages faster. However I think reproducible builds are required in the future and that also requires logs.
Additional Context
Puppetserver 8.4.0 No longer compatible with Java 11
Describe the Bug
Updating from pupperserver 8.3.0 to 8.4.0 on RHEL8 running java 11 results in service startup failure
Expected Behavior
Our two puppetservers were automatically updated last night (8.3.0 to 8.4.0) and subsequently failed to start.
Steps to Reproduce
Steps to reproduce the behavior:
- dnf update puppetserver
- systemctl status puppetserver
Environment
- Version 8.4.0
- Platform RHEL8
Additional Context
Performing a 'dnf downgrade puppetserver' returns the version to 8.3.0 and the service starts again.
Log for 8.4.0 show:
com/puppetlabs/puppetserver/JRubyPuppetResponse has been compiled by a more recent version of the Java Runtime (class file version 55.0), this version of the Java Runtime only recognizes class file versions up to 52.0
Java Support at 'https://www.puppet.com/docs/puppet/8/server/install_from_packages.html' shows Java 11 as being supported. So either this needs fixing or the documentation needs to be updated.
FIPS 140-3 Support RHEL 9
Use Case
For government use, puppetserver needs to operate in fips mode for rhel 9. This would impact the customer base. Mainly, I would like to know a timeline for fips 140-3 support so I can talk to my engineers about incorporating it into our environment.
Describe Alternatives You've Considered
Turning off Fips. Main reason I think that is not a permanent workaround is most government customers want to use puppet to improve their scores, but I think would he scared off by the fips issues.
puppetserver fails on init with >2TB of total host RAM
Describe the Bug
2024-05-07T10:36:11.458Z ERROR [async-dispatch-2] [p.t.internal] Error during service init!!!
java.lang.NumberFormatException: For input string: "2319453408"
at java.base/java.lang.NumberFormatException.forInputString(NumberFormatException.java:67)
at java.base/java.lang.Integer.parseInt(Integer.java:668)
at java.base/java.lang.Integer.parseInt(Integer.java:786)
at puppetlabs.services.master.master_core$validate_memory_requirements_BANG_.invokeStatic(master_core.clj:1235)
at puppetlabs.services.master.master_core$validate_memory_requirements_BANG_.invoke(master_core.clj:1227)
$ cat /proc/meminfo
MemTotal: 2319453408 kB
Expected Behavior
Expected fix for bigger than max Integer (2147483647) mem-size values
Steps to Reproduce
Steps to reproduce the behavior:
- setup host/env with >2TB total RAM in /proc/meminfo
- start puppetserver
Environment
- puppetserver 8.6.0
- Ubuntu 22.04
puppetserver gets trapped in exception cycle and uses 100% CPU
Describe the Bug
Puppetserver encounters an exception (outlined below) and becomes trapped in a repetitive cycle, unable to recover autonomously. To restore normal functionality, a restart is required. During this loop, it continuously logs the exception, consuming 100% of the CPU resources available. This issue occurs sporadically, with no discernible pattern evident at present.
2024-03-28T12:00:21.123+01:00 ERROR [clojure-agent-send-off-pool-83168] [p.t.s.s.status-core] #error {
:cause nil
:via
[{:type java.util.concurrent.CancellationException
:message nil
:at [java.util.concurrent.FutureTask report FutureTask.java 121]}]
:trace
[[java.util.concurrent.FutureTask report FutureTask.java 121]
[java.util.concurrent.FutureTask get FutureTask.java 191]
[clojure.core$deref_future invokeStatic core.clj 2317]
[clojure.core$future_call$reify__8544 deref core.clj 7041]
[clojure.core$deref invokeStatic core.clj 2337]
[clojure.core$deref invoke core.clj 2323]
[puppetlabs.trapperkeeper.services.status.status_core$fn__28557$guarded_status_fn_call__28562$fn__28563$fn__28573 invoke status_core.clj 377]
[puppetlabs.trapperkeeper.services.status.status_core$fn__28557$guarded_status_fn_call__28562$fn__28563 invoke status_core.clj 377]
[puppetlabs.trapperkeeper.services.status.status_core$fn__28557$guarded_status_fn_call__28562 invoke status_core.clj 359]
[puppetlabs.trapperkeeper.services.status.status_core$fn__28659$call_status_fn_for_service__28668$fn__28671 invoke status_core.clj 439]
[puppetlabs.trapperkeeper.services.status.status_core$fn__28659$call_status_fn_for_service__28668 invoke status_core.clj 421]
[puppetlabs.trapperkeeper.services.status.status_core$fn__28659$call_status_fn_for_service__28668$fn__28669 invoke status_core.clj 432]
[puppetlabs.trapperkeeper.services.status.status_core$fn__28659$call_status_fn_for_service__28668 invoke status_core.clj 421]
[puppetlabs.trapperkeeper.services.status.status_core$fn__28697$call_status_fns__28702$fn__28703$fn__28705 invoke status_core.clj 459]
[clojure.core$pmap$fn__8552$fn__8553 invoke core.clj 7089]
[clojure.core$binding_conveyor_fn$fn__5823 invoke core.clj 2047]
[clojure.lang.AFn call AFn.java 18]
[java.util.concurrent.FutureTask run FutureTask.java 264]
[java.util.concurrent.ThreadPoolExecutor runWorker ThreadPoolExecutor.java 1128]
[java.util.concurrent.ThreadPoolExecutor$Worker run ThreadPoolExecutor.java 628]
[java.lang.Thread run Thread.java 829]]}
2024-03-28T12:00:21.180+01:00 ERROR [clojure-agent-send-off-pool-82816] [p.t.s.s.status-core] Status callback for puppet-profiler timed out, shutting down background task
Environment
- Version puppetserver:
- PE: 2021.7.7.16-1jammy
- OSS: 7.16.0-1jammy
- Platform Ubuntu 22.04
Topscope variable is empty in (sub) modules
Describe the Bug
Topscope variable is empty is (sub) modules / classes, while the same variable IS known in the toplevel 00.pp
We have an 00.pp manifest files that sets a couple of top scope variables.
This 00.pp is executed for all agents.
We also have other manifests per server who offcourse call other classes.
00.pp
if '<value>' == $::facts.get('<value>') {
$az_environment = 'prod'
$test2 = "test2"
}
$test1 = "test1"
notify {"toplevel scope_test1 : ${::test1}" :}
notify {"toplevel scope_test2 : ${::test2}" :}
test.pp
class test {
notify {"test class scope_test1 : ${::test1}" :}
notify {"test class scope_test2 : ${::test2}" :}
}
When we apply (and the if statement = "True" ) , we get the results
Notice: toplevel scope_test1 : test1
Notice: toplevel scope_test2 : test2
Notice: test class scope_test1 :
Notice: test class scope_test2 : test2
Expected Behavior
I expect the top scope variable test1 to always a have the defined value, since it's defined in the top scope.
Also remarkable is that when the variable test2 is within the "if" statement , the value works as expected.
Moving the variable outside of the "if" statement , makes it "empty" in the sub modules / classes.
Environment
We run puppetserver 7.17.0 and client versions 6.28.0 / 7.31.0.
Additional Context
I noticed this issue:
https://www.puppet.com/docs/puppet/7/known_issues_puppet#pup-11437
In Puppet 6.26 and 7.14, the lookup command fails to resolve toplevel facts in hiera configs if you're using the --environment option
We run a higher version already. (7.17.0)
/status/v1/simple timing out regularly
Describe the Bug
Queries to /status/v1/simple usually return in less than 1 second, but at times they do not return at all and timeout.
I assume a timeout is a catastrophic service check, but it would be nice to get some insight into what is happening here. It seems timeouts happen even when the service is responding to requests in a healthy manner.
Expected Behavior
Rapid and accurate responses according to the state of the service and stack for load balancers and state tracking to monitor.
Steps to Reproduce
- Start puppetserver 7.x stack
- Send traffic to it (in my case, 3 clients connecting per second)
- Keep checking
/status/v1/simple - Notice that at times it just won't respond, other times the responses can take 10s of seconds or more
- Also notice that regardless of the reported health, the service is fine, even if the healthcheck times out.
Environment
- Version puppetserver 7.16.0
- Platform:
- Dockerized Ubuntu 22.04
- Custom docker image using deb from apt.puppetlabs.com with openjdk-jre-17-headless
Ubuntu 24.04 packages are missing
Use Case
Please provide packages for Ubuntu 24.04.
Describe the Solution You Would Like
A clear and concise description of what you want to happen.
Describe Alternatives You've Considered
A clear and concise description of any alternative solutions or features you've considered.
Additional Context
Add any other context or screenshots about the feature request here.
Don't open issue queue on gh if you take em in JIRA
It's confusing
Puppetserver CA API race-condition(?)
Describe the Bug
Puppetserver CA API gets into a race-condition sometimes when the Certificate Status endpoint is used to first revoke and afterwards clean the certificate. When this occurs puppetserver will start throwing ERROR [p.r.core] Internal Server Error: java.io.FileNotFoundException: /path/to/cert.pem (No such file or directory) when attempts to revoke/clean said certificate occurs.
We have not seen this bug in 7.13.0 and earlier versions. First time it occurred was after updating to 8.4.0.
Expected Behavior
Certificate is successfully revoked and cleaned and can be re-used.
Steps to Reproduce
Steps to reproduce the behavior:
- PUT /puppet-ca/v1/certificate_status/mycertname
- DELETE /puppet-ca/v1/certificate_status/mycertname
- GET /puppet-ca/v1/certificate_status/mycertname returns
ERROR [p.r.core] Internal Server Error: java.io.FileNotFoundException: /path/to/cert.pem (No such file or directory)
As stated earlier. This does not always happen.
Environment
- puppetserver 8.6.0
- Ubuntu 22.04
- CA directory is stored in glusterfs
- Container image used with extra customizations
- Modifications (both in 7.x and 8.x):
-- puppet.conf - autosign set to run a script
-- auth.conf - allow a specific certificate to use endpoint - /puppet-ca/v1/certificate_status - Modifications (new for 8.x):
-- puppetserver.conf - ca-ttl set: 5y
-- puppetserver.conf - allow-auto-renewal: true
-- puppetserver.conf - auto-renewal-cert-ttl: 5y
Additional Context
It seems to help to have the node reach out to the CA and have a new certificate request recreated and then use API to sign it. Afterwards it works again as expected.
puppetlabs/puppetserver-ca-cli#120 migh be related.
Example logs:
19/Apr/2024:11:33:35 +0000 "GET /puppet-ca/v1/certificate_status/client.domain.tld HTTP/1.1" 200 932 10.0.8.41 10.0.8.41 8140 18
2024-04-19 11:33:36,062 INFO [p.p.certificate-authority] Entity revoker.domain.tld revoked 1 certificate: client.domain.tld.
19/Apr/2024:11:33:36 +0000 "PUT /puppet-ca/v1/certificate_status/client.domain.tld HTTP/1.1" 204 0 10.0.8.41 10.0.8.41 8140 48
2024-04-19 11:33:36,073 WARN [p.p.certificate-authority] No certificate request for client.domain.tld at expected path /etc/puppetlabs/puppetserver/ca/requests/client.domain.tld.pem
19/Apr/2024:11:33:36 +0000 "DELETE /puppet-ca/v1/certificate_status/client.domain.tld HTTP/1.1" 204 162 10.0.8.41 10.0.8.41 8140 5
2024-04-19 11:33:36,315 INFO [o.e.j.u.s.SslContextFactory] x509=X509@5e09380f(private key,h=[puppet, puppet.domain.tld, puppetca, puppetca.domain.tld],a=[],w=[]) for InternalSslContextFactory@56114dcd[provider=null,keyStore=null,trustStore=null]
2024-04-19 11:34:34,070 ERROR [p.r.core] Internal Server Error: java.io.FileNotFoundException: /etc/puppetlabs/puppetserver/ca/requests/client.domain.tld.pem (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at clojure.java.io$fn__11617.invokeStatic(io.clj:229)
at clojure.java.io$fn__11617.invoke(io.clj:229)
at clojure.java.io$fn__11569$G__11523__11576.invoke(io.clj:69)
at clojure.java.io$fn__11629.invokeStatic(io.clj:258)
at clojure.java.io$fn__11629.invoke(io.clj:254)
at clojure.java.io$fn__11569$G__11523__11576.invoke(io.clj:69)
at clojure.java.io$fn__11591.invokeStatic(io.clj:165)
at clojure.java.io$fn__11591.invoke(io.clj:165)
at clojure.java.io$fn__11530$G__11519__11537.invoke(io.clj:69)
at clojure.java.io$reader.invokeStatic(io.clj:102)
at clojure.java.io$reader.doInvoke(io.clj:86)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at puppetlabs.ssl_utils.core$fn__21975$pem__GT_csr__21980$fn__21981.invoke(core.clj:727)
at puppetlabs.ssl_utils.core$fn__21975$pem__GT_csr__21980.invoke(core.clj:721)
at puppetlabs.puppetserver.certificate_authority$fn__39741$get_cert_or_csr_status__39746$fn__39750.invoke(certificate_authority.clj:2062)
at puppetlabs.puppetserver.certificate_authority$fn__39741$get_cert_or_csr_status__39746.invoke(certificate_authority.clj:2051)
at puppetlabs.services.ca.certificate_authority_core$certificate_status$fn__42391$fn__42408.invoke(certificate_authority_core.clj:471)
at liberator.core$run_handler.invokeStatic(core.clj:176)
at liberator.core$run_handler.invoke(core.clj:131)
at liberator.core$handle_ok.invokeStatic(core.clj:224)
at liberator.core$handle_ok.invoke(core.clj:224)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$multiple_representations_QMARK_.invokeStatic(core.clj:232)
at liberator.core$multiple_representations_QMARK_.invoke(core.clj:232)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$put_to_existing_QMARK_.invokeStatic(core.clj:305)
at liberator.core$put_to_existing_QMARK_.invoke(core.clj:305)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$post_to_existing_QMARK_.invokeStatic(core.clj:308)
at liberator.core$post_to_existing_QMARK_.invoke(core.clj:308)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$method_patch_QMARK_.invokeStatic(core.clj:315)
at liberator.core$method_patch_QMARK_.invoke(core.clj:315)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$method_delete_QMARK_.invokeStatic(core.clj:317)
at liberator.core$method_delete_QMARK_.invoke(core.clj:317)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$if_modified_since_exists_QMARK_.invokeStatic(core.clj:337)
at liberator.core$if_modified_since_exists_QMARK_.invoke(core.clj:337)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$if_none_match_exists_QMARK_.invokeStatic(core.clj:355)
at liberator.core$if_none_match_exists_QMARK_.invoke(core.clj:355)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$if_unmodified_since_exists_QMARK_.invokeStatic(core.clj:375)
at liberator.core$if_unmodified_since_exists_QMARK_.invoke(core.clj:375)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$if_match_exists_QMARK_.invokeStatic(core.clj:389)
at liberator.core$if_match_exists_QMARK_.invoke(core.clj:389)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$exists_QMARK_.invokeStatic(core.clj:392)
at liberator.core$exists_QMARK_.invoke(core.clj:392)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$processable_QMARK_.invokeStatic(core.clj:395)
at liberator.core$processable_QMARK_.invoke(core.clj:395)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$accept_encoding_exists_QMARK_.invokeStatic(core.clj:416)
at liberator.core$accept_encoding_exists_QMARK_.invoke(core.clj:416)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$accept_charset_exists_QMARK_.invokeStatic(core.clj:429)
at liberator.core$accept_charset_exists_QMARK_.invoke(core.clj:429)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$accept_language_exists_QMARK_.invokeStatic(core.clj:443)
at liberator.core$accept_language_exists_QMARK_.invoke(core.clj:443)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$accept_exists_QMARK_.invokeStatic(core.clj:456)
at liberator.core$accept_exists_QMARK_.invoke(core.clj:456)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$is_options_QMARK_.invokeStatic(core.clj:473)
at liberator.core$is_options_QMARK_.invoke(core.clj:473)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$valid_entity_length_QMARK_.invokeStatic(core.clj:476)
at liberator.core$valid_entity_length_QMARK_.invoke(core.clj:476)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$known_content_type_QMARK_.invokeStatic(core.clj:479)
at liberator.core$known_content_type_QMARK_.invoke(core.clj:479)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$valid_content_header_QMARK_.invokeStatic(core.clj:481)
at liberator.core$valid_content_header_QMARK_.invoke(core.clj:481)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$allowed_QMARK_.invokeStatic(core.clj:484)
at liberator.core$allowed_QMARK_.invoke(core.clj:484)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$authorized_QMARK_.invokeStatic(core.clj:487)
at liberator.core$authorized_QMARK_.invoke(core.clj:487)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$malformed_QMARK_.invokeStatic(core.clj:490)
at liberator.core$malformed_QMARK_.invoke(core.clj:490)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$method_allowed_QMARK_.invokeStatic(core.clj:493)
at liberator.core$method_allowed_QMARK_.invoke(core.clj:493)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$uri_too_long_QMARK_.invokeStatic(core.clj:496)
at liberator.core$uri_too_long_QMARK_.invoke(core.clj:496)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$known_method_QMARK_.invokeStatic(core.clj:499)
at liberator.core$known_method_QMARK_.invoke(core.clj:499)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$service_available_QMARK_.invokeStatic(core.clj:502)
at liberator.core$service_available_QMARK_.invoke(core.clj:502)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$initialize_context.invokeStatic(core.clj:504)
at liberator.core$initialize_context.invoke(core.clj:504)
at liberator.core$run_resource.invokeStatic(core.clj:595)
at liberator.core$run_resource.invoke(core.clj:593)
at puppetlabs.services.ca.certificate_authority_core$certificate_status$fn__42391.invoke(certificate_authority_core.clj:409)
at compojure.response$fn__17255.invokeStatic(response.clj:33)
at compojure.response$fn__17255.invoke(response.clj:21)
at compojure.response$fn__17228$G__17223__17235.invoke(response.clj:6)
at puppetlabs.services.ca.certificate_authority_core$fn__42441$web_routes__42446$fn__42447$fn__42448.invoke(certificate_authority_core.clj:548)
at bidi.ring$fn__17042.invokeStatic(ring.cljc:25)
at bidi.ring$fn__17042.invoke(ring.cljc:21)
at bidi.ring$fn__17027$G__17022__17036.invoke(ring.cljc:16)
at puppetlabs.comidi$make_handler$fn__18958.invoke(comidi.clj:245)
at puppetlabs.trapperkeeper.authorization.ring_middleware$fn__25846$wrap_authorization_check__25851$fn__25852$fn__25853.invoke(ring_middleware.clj:293)
at puppetlabs.ring_middleware.core$fn__23328$wrap_bad_request__23337$fn__23340$fn__23346.invoke(core.clj:187)
at puppetlabs.puppetserver.ringutils$fn__36917$wrap_with_trapperkeeper_or_client_whitelist_authorization__36922$fn__36923$fn__36927.invoke(ringutils.clj:131)
at puppetlabs.i18n.core$locale_negotiator$fn__4728.invoke(core.clj:361)
at puppetlabs.ring_middleware.core$fn__23426$wrap_uncaught_errors__23435$fn__23438$fn__23443.invoke(core.clj:233)
at puppetlabs.puppetserver.ringutils$wrap_with_puppet_version_header$fn__36906.invoke(ringutils.clj:90)
at puppetlabs.ring_middleware.core$fn__23025$wrap_response_logging__23030$fn__23031$fn__23032.invoke(core.clj:53)
at puppetlabs.trapperkeeper.services.webserver.jetty10_core$ring_handler$fn__29347.invoke(jetty10_core.clj:533)
at puppetlabs.trapperkeeper.services.webserver.jetty10_core.proxy$org.eclipse.jetty.server.handler.HandlerWrapper$ff19274a.handle(Unknown Source)
at jdk.internal.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102)
at puppetlabs.trapperkeeper.services.webserver.normalized_uri_helpers$fn__28883$normalize_uri_handler__28888$fn__28889$fn__28890.invoke(normalized_uri_helpers.clj:73)
at puppetlabs.trapperkeeper.services.webserver.normalized_uri_helpers.proxy$org.eclipse.jetty.server.handler.HandlerWrapper$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:192)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
at com.puppetlabs.trapperkeeper.services.webserver.jetty10.utils.MDCRequestLogHandler.handle(MDCRequestLogHandler.java:48)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.Server.handle(Server.java:563)
at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
at java.base/java.lang.Thread.run(Thread.java:840)
19/Apr/2024:11:34:34 +0000 "GET /puppet-ca/v1/certificate_status/client.domain.tld HTTP/1.1" 500 163 10.0.8.41 10.0.8.41 8140 18
2024-04-19 11:34:46,538 ERROR [p.r.core] Internal Server Error: java.io.FileNotFoundException: /etc/puppetlabs/puppetserver/ca/requests/client.domain.tld.pem (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at clojure.java.io$fn__11617.invokeStatic(io.clj:229)
at clojure.java.io$fn__11617.invoke(io.clj:229)
at clojure.java.io$fn__11569$G__11523__11576.invoke(io.clj:69)
at clojure.java.io$fn__11629.invokeStatic(io.clj:258)
at clojure.java.io$fn__11629.invoke(io.clj:254)
at clojure.java.io$fn__11569$G__11523__11576.invoke(io.clj:69)
at clojure.java.io$fn__11591.invokeStatic(io.clj:165)
at clojure.java.io$fn__11591.invoke(io.clj:165)
at clojure.java.io$fn__11530$G__11519__11537.invoke(io.clj:69)
at clojure.java.io$reader.invokeStatic(io.clj:102)
at clojure.java.io$reader.doInvoke(io.clj:86)
at clojure.lang.RestFn.invoke(RestFn.java:410)
at puppetlabs.ssl_utils.core$fn__21975$pem__GT_csr__21980$fn__21981.invoke(core.clj:727)
at puppetlabs.ssl_utils.core$fn__21975$pem__GT_csr__21980.invoke(core.clj:721)
at puppetlabs.puppetserver.certificate_authority$fn__39741$get_cert_or_csr_status__39746$fn__39750.invoke(certificate_authority.clj:2062)
at puppetlabs.puppetserver.certificate_authority$fn__39741$get_cert_or_csr_status__39746.invoke(certificate_authority.clj:2051)
at puppetlabs.services.ca.certificate_authority_core$certificate_status$fn__42391$fn__42408.invoke(certificate_authority_core.clj:471)
at liberator.core$run_handler.invokeStatic(core.clj:176)
at liberator.core$run_handler.invoke(core.clj:131)
at liberator.core$handle_ok.invokeStatic(core.clj:224)
at liberator.core$handle_ok.invoke(core.clj:224)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$multiple_representations_QMARK_.invokeStatic(core.clj:232)
at liberator.core$multiple_representations_QMARK_.invoke(core.clj:232)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$put_to_existing_QMARK_.invokeStatic(core.clj:305)
at liberator.core$put_to_existing_QMARK_.invoke(core.clj:305)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$post_to_existing_QMARK_.invokeStatic(core.clj:308)
at liberator.core$post_to_existing_QMARK_.invoke(core.clj:308)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$method_patch_QMARK_.invokeStatic(core.clj:315)
at liberator.core$method_patch_QMARK_.invoke(core.clj:315)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$method_delete_QMARK_.invokeStatic(core.clj:317)
at liberator.core$method_delete_QMARK_.invoke(core.clj:317)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$if_modified_since_exists_QMARK_.invokeStatic(core.clj:337)
at liberator.core$if_modified_since_exists_QMARK_.invoke(core.clj:337)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$if_none_match_exists_QMARK_.invokeStatic(core.clj:355)
at liberator.core$if_none_match_exists_QMARK_.invoke(core.clj:355)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$if_unmodified_since_exists_QMARK_.invokeStatic(core.clj:375)
at liberator.core$if_unmodified_since_exists_QMARK_.invoke(core.clj:375)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$if_match_exists_QMARK_.invokeStatic(core.clj:389)
at liberator.core$if_match_exists_QMARK_.invoke(core.clj:389)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$exists_QMARK_.invokeStatic(core.clj:392)
at liberator.core$exists_QMARK_.invoke(core.clj:392)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$processable_QMARK_.invokeStatic(core.clj:395)
at liberator.core$processable_QMARK_.invoke(core.clj:395)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$accept_encoding_exists_QMARK_.invokeStatic(core.clj:416)
at liberator.core$accept_encoding_exists_QMARK_.invoke(core.clj:416)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$accept_charset_exists_QMARK_.invokeStatic(core.clj:429)
at liberator.core$accept_charset_exists_QMARK_.invoke(core.clj:429)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$accept_language_exists_QMARK_.invokeStatic(core.clj:443)
at liberator.core$accept_language_exists_QMARK_.invoke(core.clj:443)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$accept_exists_QMARK_.invokeStatic(core.clj:456)
at liberator.core$accept_exists_QMARK_.invoke(core.clj:456)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$is_options_QMARK_.invokeStatic(core.clj:473)
at liberator.core$is_options_QMARK_.invoke(core.clj:473)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$valid_entity_length_QMARK_.invokeStatic(core.clj:476)
at liberator.core$valid_entity_length_QMARK_.invoke(core.clj:476)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$known_content_type_QMARK_.invokeStatic(core.clj:479)
at liberator.core$known_content_type_QMARK_.invoke(core.clj:479)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$valid_content_header_QMARK_.invokeStatic(core.clj:481)
at liberator.core$valid_content_header_QMARK_.invoke(core.clj:481)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$allowed_QMARK_.invokeStatic(core.clj:484)
at liberator.core$allowed_QMARK_.invoke(core.clj:484)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$authorized_QMARK_.invokeStatic(core.clj:487)
at liberator.core$authorized_QMARK_.invoke(core.clj:487)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$malformed_QMARK_.invokeStatic(core.clj:490)
at liberator.core$malformed_QMARK_.invoke(core.clj:490)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$method_allowed_QMARK_.invokeStatic(core.clj:493)
at liberator.core$method_allowed_QMARK_.invoke(core.clj:493)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$uri_too_long_QMARK_.invokeStatic(core.clj:496)
at liberator.core$uri_too_long_QMARK_.invoke(core.clj:496)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$known_method_QMARK_.invokeStatic(core.clj:499)
at liberator.core$known_method_QMARK_.invoke(core.clj:499)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$service_available_QMARK_.invokeStatic(core.clj:502)
at liberator.core$service_available_QMARK_.invoke(core.clj:502)
at liberator.core$decide.invokeStatic(core.clj:87)
at liberator.core$decide.invoke(core.clj:74)
at liberator.core$initialize_context.invokeStatic(core.clj:504)
at liberator.core$initialize_context.invoke(core.clj:504)
at liberator.core$run_resource.invokeStatic(core.clj:595)
at liberator.core$run_resource.invoke(core.clj:593)
at puppetlabs.services.ca.certificate_authority_core$certificate_status$fn__42391.invoke(certificate_authority_core.clj:409)
at compojure.response$fn__17255.invokeStatic(response.clj:33)
at compojure.response$fn__17255.invoke(response.clj:21)
at compojure.response$fn__17228$G__17223__17235.invoke(response.clj:6)
at puppetlabs.services.ca.certificate_authority_core$fn__42441$web_routes__42446$fn__42447$fn__42448.invoke(certificate_authority_core.clj:548)
at bidi.ring$fn__17042.invokeStatic(ring.cljc:25)
at bidi.ring$fn__17042.invoke(ring.cljc:21)
at bidi.ring$fn__17027$G__17022__17036.invoke(ring.cljc:16)
at puppetlabs.comidi$make_handler$fn__18958.invoke(comidi.clj:245)
at puppetlabs.trapperkeeper.authorization.ring_middleware$fn__25846$wrap_authorization_check__25851$fn__25852$fn__25853.invoke(ring_middleware.clj:293)
at puppetlabs.ring_middleware.core$fn__23328$wrap_bad_request__23337$fn__23340$fn__23346.invoke(core.clj:187)
at puppetlabs.puppetserver.ringutils$fn__36917$wrap_with_trapperkeeper_or_client_whitelist_authorization__36922$fn__36923$fn__36927.invoke(ringutils.clj:131)
at puppetlabs.i18n.core$locale_negotiator$fn__4728.invoke(core.clj:361)
at puppetlabs.ring_middleware.core$fn__23426$wrap_uncaught_errors__23435$fn__23438$fn__23443.invoke(core.clj:233)
at puppetlabs.puppetserver.ringutils$wrap_with_puppet_version_header$fn__36906.invoke(ringutils.clj:90)
at puppetlabs.ring_middleware.core$fn__23025$wrap_response_logging__23030$fn__23031$fn__23032.invoke(core.clj:53)
at puppetlabs.trapperkeeper.services.webserver.jetty10_core$ring_handler$fn__29347.invoke(jetty10_core.clj:533)
at puppetlabs.trapperkeeper.services.webserver.jetty10_core.proxy$org.eclipse.jetty.server.handler.HandlerWrapper$ff19274a.handle(Unknown Source)
at jdk.internal.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:167)
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:102)
at puppetlabs.trapperkeeper.services.webserver.normalized_uri_helpers$fn__28883$normalize_uri_handler__28888$fn__28889$fn__28890.invoke(normalized_uri_helpers.clj:73)
at puppetlabs.trapperkeeper.services.webserver.normalized_uri_helpers.proxy$org.eclipse.jetty.server.handler.HandlerWrapper$ff19274a.handle(Unknown Source)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:192)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)
at com.puppetlabs.trapperkeeper.services.webserver.jetty10.utils.MDCRequestLogHandler.handle(MDCRequestLogHandler.java:48)
at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
at org.eclipse.jetty.server.Server.handle(Server.java:563)
at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558)
at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379)
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
at java.base/java.lang.Thread.run(Thread.java:840)
19/Apr/2024:11:34:46 +0000 "GET /puppet-ca/v1/certificate_status/client.domain.tld HTTP/1.1" 500 163 10.0.8.41 10.0.8.41 8140 16
/status/v1/services only reports running status
Describe the Bug
According to API docs here https://www.puppet.com/docs/puppet/7/server/status-api/v1/services#get-statusv1services the status should return multiple different states, but it seems only running is ever returned.
When querying the services API, state running is returned after only 1 jruby instance is ready, not starting. I've never been able to get starting to show. I'm running 31 jruby instances.
Expected Behavior
All the states should be exposed according to the actual state of the service.
Steps to Reproduce
- Configure more than 1 jruby instance in the config
- Start puppetserver
- Query the API while watching the logfile
- It will take 1-2 seconds for each jruby instance to start until max instances are reached.
- During this time it is expected that state would be
startingbutrunningis returned instead.
Environment
- Version puppetserver 7.16.0
- Platform:
- Dockerized Ubuntu 22.04
- Custom docker image using deb from apt.puppetlabs.com with openjdk-jre-17-headless
Update concurrent-ruby to latest
Issue
Puppet Agent removed a workaround for concurrent-ruby < 1.1.9 that is required to prevent request threads from leaking memory. This was intended to be paired with upgrading the version of concurrent-ruby shipped across the product to 1.1.10. However, because of packaging constraints some products were upgraded to 1.1.9 and some to 1.1.10, and in the confusion Puppet Server wasn't upgraded at all.
This affects users running Puppet Server with versions of the Puppet Agent from 7.25.0 & 8.1.0 onwards. The Puppet Server version doesn't matter, but those versions of the Puppet Agent were shipped in Platform and PE releases from this summer and autumn.
Describe the Solution You Would Like
Puppet Server should not leak memory.
To resolve the issue we should update concurrent-ruby to be inline with the versions shipped in the platform. At least 1.1.10, and preferably latest (1.2.2 at this time).
Describe Alternatives You've Considered
According to the concurrent-ruby maintainers the issue should be resolved post 1.1.9, however the underlying structure that was doing the leaking remains until a major refactor removed it in 1.2.0. There's some concern that we may still leak objects until that major refactor. However, we don't have conclusive evidence either way. We had done some tests that showed 1.1.10 did not leak, but since this issued escaped, we will consider that evidence inconclusive until we do an RCA.
Alternatively, we could revert the Puppet Agent change, but that change is required to take up 1.2.0+ of concurrent-ruby.
Additional Context & Workaround
- Users may resolve this issue on deployed systems by manually updating the concurrent-ruby gem, either through the package type and the puppetserver_gem provider, or by running
puppetserver gem install --no-document -v 1.2.2 concurrent-ruby. - Users should not need to remove the older version of concurrent-ruby, but if they wish to they can run
gem uninstall -i /opt/puppetlabs/server/data/puppetserver/vendored-jruby-gems concurrent-rubyto do so. - We need to have this fix in all of our products' source trees before the holidays for releases early next year.
- Bolt was the primary reason for only upgrading to 1.1.9 in part of the platform, and Puppet itself was only tested with 1.1.10, so we need to make sure that Bolt's reason for not upgrading was merely conservative packaging pins and that neither Bolt nor Puppet usage of concurrent-ruby break on an upgrade to 1.2.2 (latest). - Initial code review doesn't show any API breakages, and initial testing hasn't discovered any issues, but we are continuing on both fronts.
- See puppetlabs/puppet-runtime#775 for discussion on what versions go in which branch.
Provide more detail about service state during startup/jruby instance creation
/status/v1/services seems to return running state once the first jruby instance is started. I'd prefer to wait until all of my instances have started before exposing that the service is healthy. 1 instance cannot handle the flood of traffic soon to come its way.
/status/v1/services?level=debug gives more detail including:
num-free-jrubies, but if there is already traffic then this count will be lower than the available jrubies.num-jrubiesseems to be the total configured to run, equivalent tomax-active-instancesinpuppetserver.conf
Maybe if another metric could be exposed here of num-initialized-jrubies or equivalent, healthcheck could monitor that. And maybe also a way for the basic /status/v1/services to return startup instead of running until num-initialized-jrubies == num-jrubies while in a startup mode.
Are there currently any ways to prevent traffic until the service is ready regarding these contexts?
Missing Puppet packages for Debian 12 (Bookworm)
Hello!
Debian 12 AIO packages for Puppet-Agent are now available (see puppetlabs/puppet#9149).
We now need PuppetServer (this issue) and PuppetDB (puppetlabs/puppetdb#3950) packages for this operating system 😁.
Puppet Server does not update CRLs that are close to expiring
Expiration of the Certificate Revocation List (CRL) is fatal to
communication between Puppet Enterprise components, resulting
in a complete outage of service. Puppet 8 sets the crl_refresh_interval
to 1 day by default so that agents will pull in updates to the
CRL file.
However, Puppet Server does not ensure CRL entries are updated
on a regular cadence. In most installations, there is some
level of turnover in the agent population which results in
CRL updates. But, PE enables the infrastructure CRL which
is only updated by the addition or removal of a compiler node.
Additionally, Puppet 6 adds a "Root CA" with an associated CRL
for which no update workflow exists.
Without automated updates to ensure CRLs are refreshed,
every Puppet installation is at risk of a complete outage
when this component expires.
Reproduction Case
Obtain a RHEL 8 VM.
Install PE 2021.7.2.
Ensure CRL refresh is enabled:
/opt/puppetlabs/bin/puppet config set crl_refresh_interval 1d
Create and destroy a certificate to update leaf CRLs with a 5 year expiration:
# Stop puppet agent to prevent management of infra_inventory.txt
systemctl stop puppet
/opt/puppetlabs/bin/puppetserver ca generate --certname foo.example
printf '\nfoo.example\n' >> /etc/puppetlabs/puppetserver/ca/infra_inventory.txt
/opt/puppetlabs/bin/puppetserver ca clean --certname foo.example
Disable clock synchronization and then set the system forward to within
30 days of CRL expiration:
timedatectl set-ntp false
# Additionally, if VM is hosted by vSphere
vmware-toolbox-cmd timesync disable
# Check CRL expiration. Currently hard-coded to 5 years for CRLs generated
# by the Puppet Server process.
openssl crl -in "$(puppet config print cacrl)" -noout -nextupdate
timedatectl set-time "$(date --date '1800 days' +'%Y-%m-%d %H:%M:%S')"
Re-start Puppet Server and run the agent:
systemctl restart pe-puppetserver
puppet agent -t
Advance the system clock another 30 days and run the agent:
timedatectl set-time "$(date --date '30 days' +'%Y-%m-%d %H:%M:%S')"
puppet agent -t
Outcome
The agent run fails due to an expired CRL:
# puppet agent -t
Info: Refreshing CRL
Error: certificate verify failed [CRL has expired for CN=deluxe-mile.delivery.puppetlabs.net]
Error: certificate verify failed [CRL has expired for CN=deluxe-mile.delivery.puppetlabs.net]
Expected Outcome
At service start, and on a regular interval, Puppet Server updates any CRL
that is within 30 days of expiration.
The example above only presents the expiration of the leaf CRL, but the
CRL from the "Puppet Root CA" must also be considered. Puppet Server
should refresh any CRL in the chain for which it has access to the
corresponding private key.
Memory Leak when passing ca_file to Net::HTTP in a custom function
Describe the Bug
There is a memory leak when using Net::HTTP with ca_file to access a https:// URL in a custom function.
Removing the ca_file parameter, the leak is no longer visible.
Expected Behavior
No memory leak.
Steps to Reproduce
Call something like this in a manifest:
require 'net/http'
require 'uri'
module Puppet::Parser::Functions
newfunction(:foo, :type => :rvalue)) do |args|
uri = URI('https://foo.com')
response = Net::HTTP.start(uri.hostname, uri.port,
:use_ssl => true,
:verify_mode = OpenSSL::SSL::VERIFY_PEER,
:ca_file => '/etc/pki/tls/certs/ca-bundle.trust.crt',
) do |http|
http.request(request)
end
return ""
end
end
Environment
RHEL9
puppet-agent-8.6.0-1.el9.x86_64
puppetdb-8.5.0-1.el9.noarch
puppetdb-termini-8.5.0-1.el9.noarch
puppetserver-8.6.1-1.el9.noarch
openjdk 17.0.9 2023-10-17 LTS
Additional Context
I found https://bugs.ruby-lang.org/issues/15082#note-5, however I don't know if it's related.
The setup has 73 Agents, all with default runinterval. After ~25 hours it would OOM with 10GiB of JVM heap configured.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.