Comments (5)
Pinging @elastic/es-security (Team:Security)
from elasticsearch.
Muting this since I've also seen this fail earlier today: https://gradle-enterprise.elastic.co/s/skv7lgzrvktq6/tests/task/:x-pack:plugin:core:test/details/org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests/testReloadingKeyStore?top-execution=1
from elasticsearch.
I can reliable reproduce with Java 23, but not Java 22
Java.net | | 23.ea.23 | open | installed | 23.ea.23-open
To avoid some misleading warnings, rule out the security manager, and provide better logging here is a better reproduction line:
./gradlew ':x-pack:plugin:core:test' --tests "org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests.testReloadingKeyStore" -Dtests.seed=BAFBC3F9A8DA6D20 -Dtests.locale=da -Dtests.timezone=Indian/Comoro -Druntime.java=23 -Dtests.security.manager=false -Dtests.jvm.argline="-Dorg.elasticsearch.nativeaccess.enableVectorLibrary=false --enable-native-access=ALL-UNNAMED" -Dtests.es.logger.org.apache.http=debug -Dtests.es.logger.level=DEBUG
fails with Java 23, but works with Java 22.
Java 22 (works)
1> [2024-05-22T01:01:34,116][DEBUG][o.a.h.i.c.DefaultHttpClientConnectionOperator] [testReloadingKeyStore] Connection established org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests$1@371b6acc
1> [2024-05-22T01:01:34,116][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Executing request GET / HTTP/1.1
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Target auth state: UNCHALLENGED
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Proxy auth state: UNCHALLENGED
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 >> GET / HTTP/1.1
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 >> Host: localhost:52242
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 >> Connection: Keep-Alive
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.14 (Java/22)
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 >> Accept-Encoding: gzip,deflate
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "GET / HTTP/1.1[\r][\n]"
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "Host: localhost:52242[\r][\n]"
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.14 (Java/22)[\r][\n]"
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
1> [2024-05-22T01:01:34,117][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "[\r][\n]"
1> [2024-05-22T01:01:34,120][DEBUG][o.e.t.h.MockWebServer ] [[HTTP-Dispatcher]] [127.0.0.1:52242] incoming HTTP request [GET /], returning status [200] body [body]
1> [2024-05-22T01:01:34,125][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 << "HTTP/1.1 200 OK[\r][\n]"
1> [2024-05-22T01:01:34,126][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 << "Date: Tue, 21 May 2024 22:01:34 GMT[\r][\n]"
1> [2024-05-22T01:01:34,126][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 << "Content-length: 4[\r][\n]"
1> [2024-05-22T01:01:34,126][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 << "[\r][\n]"
1> [2024-05-22T01:01:34,129][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 << HTTP/1.1 200 OK
1> [2024-05-22T01:01:34,129][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 << Date: Tue, 21 May 2024 22:01:34 GMT
1> [2024-05-22T01:01:34,129][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 << Content-length: 4
1> [2024-05-22T01:01:34,130][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Connection can be kept alive indefinitely
1> [2024-05-22T01:01:34,132][DEBUG][o.a.h.i.c.DefaultManagedHttpClientConnection] [testReloadingKeyStore] http-outgoing-0: Close connection
Java 23 (fails)
1> [2024-05-22T01:04:18,096][DEBUG][o.a.h.i.c.DefaultHttpClientConnectionOperator] [testReloadingKeyStore] Connection established org.elasticsearch.xpack.core.ssl.SSLConfigurationReloaderTests$1@ade8676
1> [2024-05-22T01:04:18,096][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Executing request GET / HTTP/1.1
1> [2024-05-22T01:04:18,096][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Target auth state: UNCHALLENGED
1> [2024-05-22T01:04:18,096][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Proxy auth state: UNCHALLENGED
1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 >> GET / HTTP/1.1
1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 >> Host: localhost:52320
1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 >> Connection: Keep-Alive
1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.14 (Java/23-ea)
1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.headers ] [testReloadingKeyStore] http-outgoing-0 >> Accept-Encoding: gzip,deflate
1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "GET / HTTP/1.1[\r][\n]"
1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "Host: localhost:52320[\r][\n]"
1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.14 (Java/23-ea)[\r][\n]"
1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "[\r][\n]"
1> [2024-05-22T01:04:18,097][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "[write] I/O error: Broken pipe"
1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.i.c.DefaultManagedHttpClientConnection] [testReloadingKeyStore] http-outgoing-0: Close connection
1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "GET / HTTP/1.1[\r][\n]"
1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "Host: localhost:52320[\r][\n]"
1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "Connection: Keep-Alive[\r][\n]"
1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "User-Agent: Apache-HttpClient/4.5.14 (Java/23-ea)[\r][\n]"
1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "Accept-Encoding: gzip,deflate[\r][\n]"
1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "[\r][\n]"
1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.wire ] [testReloadingKeyStore] http-outgoing-0 >> "[write] I/O error: Connection or outbound has closed"
1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.i.c.DefaultManagedHttpClientConnection] [testReloadingKeyStore] http-outgoing-0: Shutdown connection
1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.i.e.MainClientExec ] [testReloadingKeyStore] Connection discarded
1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.i.c.PoolingHttpClientConnectionManager] [testReloadingKeyStore] Connection released: [id: 0][route: {s}->https://localhost:52320][total available: 0; route allocated: 0 of 2; total allocated: 0 of 20]
1> [2024-05-22T01:04:18,098][INFO ][o.a.h.i.e.RetryExec ] [testReloadingKeyStore] I/O exception (java.net.SocketException) caught when processing request to {s}->https://localhost:52320: Broken pipe
1> [2024-05-22T01:04:18,098][DEBUG][o.a.h.i.e.RetryExec ] [testReloadingKeyStore] Broken pipe
I am pretty clueless to what the root cause may be. It is either an issue with out MockWebServer which delegates down to com.sun.net.httpserver.HttpsServer or an issue with apache http client. I ran another test that uses the MockWebServer with HTTPS and it worked..so I am pretty clueless but take another look soon.
from elasticsearch.
I can get the test to pass in Java23 by changing:
--- a/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/ssl/SSLConfigurationReloaderTests.java
+++ b/x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/ssl/SSLConfigurationReloaderTests.java
@@ -130,7 +130,7 @@ public class SSLConfigurationReloaderTests extends ESTestCase {
// Load HTTPClient only once. Client uses the same store as a truststore
try (CloseableHttpClient client = getSSLClient(keystorePath, "testnode")) {
final Consumer<SSLContext> keyMaterialPreChecks = (context) -> {
- try (MockWebServer server = new MockWebServer(context, true)) {
+ try (MockWebServer server = new MockWebServer(context, false)) {
server.enqueue(new MockResponse().setResponseCode(200).setBody("body"));
server.start();
privilegedConnect(() -> client.execute(new HttpGet("https://localhost:" + server.getPort())).close());
So, the problem with the test is that in Java23 we are wiring up an apache http client to a mock web server with mutual TLS and for some reason the mock web server does not trust the apache http client.
It isn't clear if mTLS is intentional with this test since it really is not an important detail of what is being tested. It also not clear if it worked because the mock web server and apache http client are configured correctly for mTLS or just happened to work in the past. We tend to conflate key and trust stores, so maybe we aren't wiring up mTLS correctly and are relying on a Java bug (pre java 23) for this to work? I'll keep chipping away, but starting to gain more confidence this is not a production concern.
from elasticsearch.
Pretty confident that for some reason i can not find that pre Java23 mTLS via the MockWebServer-> com.sun.net.httpserver.HttpsServer simply didn't work. I am also pretty sure that asking for mTLS from this mock web server is a typo since we never give the client any private credentials. I think Java23 is behaving correctly, and Java22- is wrong. In Java 22, I was able to get the mock HTTP server to return it's mock response to a cURL call that absolutely does not present a certificate to the server. The same cURL test failed to form a TLS connection in Java 23, which is the correct behavior since the mock web server explicitly requests mTLS.
To fix this, I will remove the mTLS from the test since it was never part of what the test was testing. I don't know specifically what was fixed/broken but since we don't use com.sun.net.httpserver.HttpsServer in production code, probably not a big deal.
from elasticsearch.
Related Issues (20)
- TransportSearchAction/CanMatchPreFilterSearchPhase may block transport_worker HOT 4
- Elastic search smart chinese plugin returns invalid tokens HOT 1
- [CI] SamlServiceProviderMetadataIT classMethod failing HOT 4
- [CI] NativePrivilegeStoreCacheTests testGetPrivilegesUsesCache failing HOT 3
- `date` field types fail to index specific malformed data even with `ignore_malformed` enabled HOT 3
- ESQL: Refactor Lookup to have the lookup tables modeled into a Relation in the pre-analysis stage HOT 1
- Date range query returns different results in ES7 vs. ES8 - does not respect include_lower HOT 2
- Upgrade from 7.x to 8.x fails if 7.x node didn't fully start up HOT 1
- Synthetic source failure when `doc_values: false` HOT 1
- [Transform] Telemetry, Metrics, and Profiling HOT 2
- [CI] TimeSeriesSortedSourceOperatorTests testMatchNone failing HOT 2
- [CI] CoreWithSecurityClientYamlTestSuiteIT test {yaml=search.vectors/180_update_dense_vector_type/Test create and update dense vector mapping with bulk indexing} failing HOT 4
- [CI] DenseVectorMappingUpdateIT testDenseVectorMappingUpdateOnOldCluster HOT 2
- [async search] add search_id, is_running and is_complete to response headers HOT 4
- [CI] TimeSeriesSortedSourceOperatorTests testMatchNone failing HOT 1
- Accept ["_none_"] to disable stored fields HOT 3
- [ML] categorize_text agg produces no results when used with a muiti-field text field HOT 2
- [CI] UpgradeWithOldIndexSettingsIT testMapperDynamicIndexSetting {cluster=UPGRADED} failing HOT 9
- Transport versioning for plugins in order to support rolling upgrades HOT 2
- The English word 'IT' cannot be filtered HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elasticsearch.