Comments (2)
Pinging @elastic/ml-core (Team:ML)
from elasticsearch.
In a 3 node cluster 2 of the nodes can't connect to the 3rd. Checking the logs the disconnected node appears to have gone in feature reset loop.
I can't see any obvious reason for the reset loop but one concern is that reset code is tying up a network thread, hence the node fell out of the cluster.
I've assigned medium risk based on the facts that this code is long established, has not been modified recently and this is the first failure of this kind I have found.
[2024-05-17T13:15:02,544][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Starting machine learning feature reset
[2024-05-17T13:15:02,622][DEBUG][o.e.x.m.a.TransportCancelJobModelSnapshotUpgradeAction] [javaRestTest-0] [_all] cancel model snapshot [_all] upgrades
[2024-05-17T13:15:02,623][DEBUG][o.e.x.m.a.TransportKillProcessAction] [javaRestTest-0] [*] Cannot kill the process because job(s) are not open
[2024-05-17T13:15:02,625][DEBUG][o.e.x.m.a.TransportStopDataFrameAnalyticsAction] [javaRestTest-0] Received request to stop data frame analytics [_all]
[2024-05-17T13:15:02,625][DEBUG][o.e.x.m.a.TransportStopDataFrameAnalyticsAction] [javaRestTest-0] Resolved data frame analytics to stop: []
[2024-05-17T13:15:02,647][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-annotations-000001/CfE6_bXRTXG_PQVrSWjV6A] deleting index
[2024-05-17T13:15:02,647][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-notifications-000002/FQgd2g8cQI6mp2S7AuugEw] deleting index
[2024-05-17T13:15:02,647][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-inference-native-000002/cWhGVDKdQcmkkTuc5s6XDQ] deleting index
[2024-05-17T13:15:02,647][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-inference-000005/brlBcFhGTPOzrQN5jzoHgw] deleting index
[2024-05-17T13:15:02,693][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Finished machine learning feature reset
[2024-05-17T13:15:02,716][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Starting machine learning feature reset
[2024-05-17T13:15:02,762][DEBUG][o.e.x.m.a.TransportCancelJobModelSnapshotUpgradeAction] [javaRestTest-0] [_all] cancel model snapshot [_all] upgrades
[2024-05-17T13:15:02,763][DEBUG][o.e.x.m.a.TransportKillProcessAction] [javaRestTest-0] [*] Cannot kill the process because job(s) are not open
[2024-05-17T13:15:02,764][DEBUG][o.e.x.m.a.TransportStopDataFrameAnalyticsAction] [javaRestTest-0] Received request to stop data frame analytics [_all]
[2024-05-17T13:15:02,765][DEBUG][o.e.x.m.a.TransportStopDataFrameAnalyticsAction] [javaRestTest-0] Resolved data frame analytics to stop: []
[2024-05-17T13:15:02,804][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Finished machine learning feature reset
[2024-05-17T13:15:02,806][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [text-expansion-pipeline-test-index/xhXxcR54TTeH_Cz-qy3jEA] deleting index
[2024-05-17T13:15:02,959][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.ml-inference-000005] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2024-05-17T13:15:03,007][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.ml-annotations-000001] creating index, cause [api], templates [], shards [1]/[1]
[2024-05-17T13:15:03,231][INFO ][o.e.c.r.a.AllocationService] [javaRestTest-0] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ml-annotations-000001][0]]])." previous.health="YELLOW" reason="shards started [[.ml-annotations-000001][0]]"
[2024-05-17T13:15:03,248][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Starting machine learning feature reset
[2024-05-17T13:15:03,321][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-annotations-000001/B1nXfB8wTj26mKIZZy5bAg] deleting index
[2024-05-17T13:15:03,321][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-inference-000005/OrAILs_nSR6LFCB2I3VwPg] deleting index
[2024-05-17T13:15:03,383][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Finished machine learning feature reset
[2024-05-17T13:15:03,409][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Starting machine learning feature reset
[2024-05-17T13:15:03,499][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Finished machine learning feature reset
[2024-05-17T13:15:03,528][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.ml-inference-000005] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2024-05-17T13:15:03,573][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.ml-annotations-000001] creating index, cause [api], templates [], shards [1]/[1]
[2024-05-17T13:15:03,770][INFO ][o.e.c.r.a.AllocationService] [javaRestTest-0] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ml-annotations-000001][0]]])." previous.health="YELLOW" reason="shards started [[.ml-annotations-000001][0]]"
[2024-05-17T13:15:03,770][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Starting machine learning feature reset
[2024-05-17T13:15:03,866][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-annotations-000001/XqfInErASiu-OZOwRLxnKA] deleting index
[2024-05-17T13:15:03,866][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-inference-000005/Ar9ILsa3QwS_JBu-cJ1PXQ] deleting index
[2024-05-17T13:15:03,908][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Finished machine learning feature reset
[2024-05-17T13:15:03,932][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Starting machine learning feature reset
[2024-05-17T13:15:04,019][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Finished machine learning feature reset
[2024-05-17T13:15:04,046][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.ml-inference-000005] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2024-05-17T13:15:04,090][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.ml-annotations-000001] creating index, cause [api], templates [], shards [1]/[1]
[2024-05-17T13:15:04,235][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.ml-inference-native-000002] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2024-05-17T13:15:04,417][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Starting machine learning feature reset
[2024-05-17T13:15:04,525][INFO ][o.e.c.r.a.AllocationService] [javaRestTest-0] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ml-inference-native-000002][0]]])." previous.health="YELLOW" reason="shards started [[.ml-inference-native-000002][0]]"
[2024-05-17T13:15:04,567][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-annotations-000001/zNhk6U9tSVKecPGrfApFIg] deleting index
[2024-05-17T13:15:04,567][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-inference-000005/6nF42COrSAS4Uw4EOv9mxQ] deleting index
[2024-05-17T13:15:04,567][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-inference-native-000002/h6wlOfd0Qy60JFPzBiT7GQ] deleting index
[2024-05-17T13:15:04,612][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Finished machine learning feature reset
[2024-05-17T13:15:04,636][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Starting machine learning feature reset
[2024-05-17T13:15:04,732][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Finished machine learning feature reset
[2024-05-17T13:15:05,040][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Starting machine learning feature reset
[2024-05-17T13:15:05,128][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Finished machine learning feature reset
[2024-05-17T13:15:05,151][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Starting machine learning feature reset
[2024-05-17T13:15:05,223][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Finished machine learning feature reset
[2024-05-17T13:15:05,271][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.ml-inference-000005] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2024-05-17T13:15:05,316][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.ml-annotations-000001] creating index, cause [api], templates [], shards [1]/[1]
[2024-05-17T13:15:05,510][INFO ][o.e.c.r.a.AllocationService] [javaRestTest-0] current.health="GREEN" message="Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[.ml-inference-000005][0], [.ml-annotations-000001][0]]])." previous.health="YELLOW" reason="shards started [[.ml-inference-000005][0], [.ml-annotations-000001][0]]"
[2024-05-17T13:15:05,531][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Starting machine learning feature reset
[2024-05-17T13:15:05,532][INFO ][o.e.c.m.MetadataCreateIndexService] [javaRestTest-0] [.ml-notifications-000002] creating index, cause [auto(bulk api)], templates [.ml-notifications-000002], shards [1]/[1]
[2024-05-17T13:15:05,674][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-annotations-000001/mtyJd8vPRyC_1_kxdtcVQg] deleting index
[2024-05-17T13:15:05,674][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-notifications-000002/CEkMpqBLQI2rARXXrSVJwQ] deleting index
[2024-05-17T13:15:05,680][INFO ][o.e.c.m.MetadataDeleteIndexService] [javaRestTest-0] [.ml-inference-000005/YQKHwEr-TCi82dFf_rIG4A] deleting index
[2024-05-17T13:15:05,751][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Finished machine learning feature reset
[2024-05-17T13:15:05,775][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Starting machine learning feature reset
[2024-05-17T13:15:05,862][INFO ][o.e.x.m.MachineLearning ] [javaRestTest-0] Finished machine learning feature reset
from elasticsearch.
Related Issues (20)
- [ESQL] Support Date Diff function on date nanos HOT 1
- [ESQL] Support Date Extract function on date nanos HOT 1
- [ESQL] Support Max & Min aggregation on Date Nanos HOT 1
- [ESQL] Support Count & Count Distinct aggregations on date nanos HOT 1
- [ESQL] Support Values aggregation on date nanos HOT 1
- [ESQL] Date Trunc function should support date nanos HOT 1
- [ESQL] Support autocasting to resolve types between nanosecond dates and millisecond dates HOT 1
- Support `major.minor` tag in docker images HOT 1
- [CI] NativePrivilegeStoreCacheTests testPopulationOfCacheWhenLoadingPrivilegesForAllApplications failing HOT 1
- About Elasticsearch License HOT 2
- [CI] IndexRecoveryIT testSourceThrottling failing HOT 1
- [ES|QL][Named parameters] Not sending the params in a query with named parameters results in blocking all future _query requests HOT 2
- [CI] LangPainlessClientYamlTestSuiteIT test {yaml=painless/140_dense_vector_basic/Test hamming distance fails on float} failing HOT 1
- [CI] RolloverIT testRolloverWithClosedWriteIndex failing HOT 1
- Support synthetic source for fields with copy_to feature enabled HOT 1
- Support synthetic source for fields with normalizers enabled HOT 1
- ESQL: a second `ENRICH` enriching with a `RENAME`d field fails HOT 1
- [CI] DataStreamsClientYamlTestSuiteIT test {p0=data_stream/200_rollover_failure_store/Rolling over a failure store on a data stream without the failure store enabled should work} failing HOT 2
- [CI] JvmCrashIT testJvmCrash failing HOT 1
- Improve zero_terms_query option HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from elasticsearch.