Comments (1)
vijaysgovind
commented on May 27, 2024
Start Run on OCI Ampere Server
output of docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
e684bb73f776 weaviate-reranker-transformers-1 0.15% 353.4MiB / 23.43GiB 1.47% 3.46kB / 4.9kB 457MB / 16.4kB 12
a8d91fc44be3 weaviate-multi2vec-bind-1 377.17% 4.899GiB / 23.43GiB 20.91% 1.1MB / 301kB 5.18GB / 45.1kB 28
4df61d029961 weaviate-weaviate-1 0.41% 65.02MiB / 23.43GiB 0.27% 1.43MB / 1.13MB 0B / 7.41MB 10
btop output
htop output
output of watch free -m
Every 2.0s: free -m govind: Sat Jan 13 22:35:04 2024
total used free shared buff/cache available
Mem: 23987 6234 10969 5 6783 17433
Swap: 0 0 0
Capturing the docker logs using the following commands
(base) ubuntu@govind:~/docker/weaviate$ docker compose logs weaviate -f > weaviate.log
(base) ubuntu@govind:~/docker/weaviate$ docker compose logs multi2vec-bind -f > multivecbind.log
(base) ubuntu@govind:~/docker/weaviate$ docker compose logs reranker-transformers -f > reranker-transformers.log
Oracle Cloud Instance Utilization
Output of docker stats during vectorization
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
e684bb73f776 weaviate-reranker-transformers-1 0.15% 353.4MiB / 23.43GiB 1.47% 3.53kB / 4.9kB 457MB / 16.4kB 12
a8d91fc44be3 weaviate-multi2vec-bind-1 296.57% 4.919GiB / 23.43GiB 21.00% 157MB / 51.2MB 5.18GB / 45.1kB 28
4df61d029961 weaviate-weaviate-1 11.92% 2.831GiB / 23.43GiB 12.09% 210MB / 159MB 434kB / 3.59GB 11
output of btop command during vectorization
Output of docker stats after ~18 hours
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
e684bb73f776 weaviate-reranker-transformers-1 0.15% 411.6MiB / 23.43GiB 1.72% 4.51kB / 4.9kB 457MB / 16.4kB 12
a8d91fc44be3 weaviate-multi2vec-bind-1 383.29% 5.114GiB / 23.43GiB 21.83% 1.24GB / 414MB 5.18GB / 45.1kB 40
4df61d029961 weaviate-weaviate-1 0.29% 10.03GiB / 23.43GiB 42.80% 1.66GB / 1.25GB 5.84GB / 67.2GB 11
A few moments later
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
e684bb73f776 weaviate-reranker-transformers-1 0.14% 326.6MiB / 23.43GiB 1.36% 4.51kB / 4.9kB 457MB / 16.4kB 12
a8d91fc44be3 weaviate-multi2vec-bind-1 258.50% 5.024GiB / 23.43GiB 21.45% 1.24GB / 414MB 5.18GB / 45.1kB 40
4df61d029961 weaviate-weaviate-1 66.91% 17.15GiB / 23.43GiB 73.22% 1.67GB / 1.25GB 15.4GB / 85.2GB 11
Status of btop
Vscode iters/seconds
18160it [15:29:09, 8.43s/it] -> At the start this is at 2.8-3 s/iters
OCI Dashboards
Batch Insert Fails at below
18164it [15:35:18, 3.09s/it]
{
"name": "WeaviateQueryException",
"message": "Query call failed with message Socket closed.",
"stack": "---------------------------------------------------------------------------
_InactiveRpcError Traceback (most recent call last)
File ~/miniconda3/lib/python3.11/site-packages/weaviate/collections/batch/grpc.py:86, in _BatchGRPC.__send_batch(self, batch)
85 res: batch_pb2.BatchObjectsReply
---> 86 res, _ = self._connection.grpc_stub.BatchObjects.with_call(
87 batch_pb2.BatchObjectsRequest(
88 objects=batch,
89 consistency_level=self._consistency_level,
90 ),
91 metadata=metadata,
92 )
94 objects: Dict[int, str] = {}
File ~/miniconda3/lib/python3.11/site-packages/grpc/_channel.py:1177, in _UnaryUnaryMultiCallable.with_call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
1171 (
1172 state,
1173 call,
1174 ) = self._blocking(
1175 request, timeout, metadata, credentials, wait_for_ready, compression
1176 )
-> 1177 return _end_unary_response_blocking(state, call, True, None)
File ~/miniconda3/lib/python3.11/site-packages/grpc/_channel.py:1003, in _end_unary_response_blocking(state, call, with_call, deadline)
1002 else:
-> 1003 raise _InactiveRpcError(state)
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
\tstatus = StatusCode.UNAVAILABLE
\tdetails = \"Socket closed\"
\tdebug_error_string = \"UNKNOWN:Error received from peer {created_time:\"2024-01-14T14:04:30.667778327+09:00\", grpc_status:14, grpc_message:\"Socket closed\"}\"
>
During handling of the above exception, another exception occurred:
WeaviateQueryException Traceback (most recent call last)
Cell In[6], line 18
16 # import images in batches of 5
17 if (len(items) > 4):
---> 18 response = foods.data.insert_many(items)
19 items.clear()
21 # Insert any remaining items
File ~/miniconda3/lib/python3.11/site-packages/weaviate/collections/data.py:375, in _DataCollection.insert_many(self, objects)
357 def insert_many(
358 self,
359 objects: List[Union[Properties, DataObject[Properties]]],
360 ) -> BatchObjectReturn:
361 \"\"\"Insert multiple objects into the collection.
362
363 Arguments:
(...)
373 If a property is invalid. I.e., has name `id` or `vector`, which are reserved.
374 \"\"\"
--> 375 return self._batch_grpc.objects(
376 [
377 _BatchObject(
378 collection=self.name,
379 vector=obj.vector,
380 uuid=obj.uuid,
381 properties=cast(dict, obj.properties),
382 tenant=self._tenant,
383 )
384 if isinstance(obj, DataObject)
385 else _BatchObject(
386 collection=self.name,
387 vector=None,
388 uuid=None,
389 properties=cast(dict, obj),
390 tenant=None,
391 )
392 for obj in objects
393 ]
394 )
File ~/miniconda3/lib/python3.11/site-packages/weaviate/collections/batch/grpc.py:54, in _BatchGRPC.objects(self, objects)
40 weaviate_objs: List[batch_pb2.BatchObject] = [
41 batch_pb2.BatchObject(
42 collection=obj.collection,
(...)
50 for obj in objects
51 ]
53 start = time.time()
---> 54 errors = self.__send_batch(weaviate_objs)
55 elapsed_time = time.time() - start
57 all_responses: List[Union[uuid_package.UUID, ErrorObject]] = cast(
58 List[Union[uuid_package.UUID, ErrorObject]], list(range(len(weaviate_objs)))
59 )
File ~/miniconda3/lib/python3.11/site-packages/weaviate/collections/batch/grpc.py:99, in _BatchGRPC.__send_batch(self, batch)
97 return objects
98 except grpc.RpcError as e:
---> 99 raise WeaviateQueryException(e.details())
WeaviateQueryException: Query call failed with message Socket closed."
}
OCI Console Screenshot
As can be seen from above there a "break" in the monitoring after the memory utilization shoots up to the 24G limit of the OCI server.
btop output after crash
free -m output
total used free shared buff/cache available
Mem: 23987 23646 171 5 169 74
Swap: 0
docker stats output
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
e684bb73f776 weaviate-reranker-transformers-1 0.14% 324.4MiB / 23.43GiB 1.35% 5.06kB / 5.36kB 6.83GB / 16.4kB 12
a8d91fc44be3 weaviate-multi2vec-bind-1 286.45% 5.006GiB / 23.43GiB 21.37% 1.24GB / 414MB 18.8GB / 45.1kB 40
4df61d029961 weaviate-weaviate-1 104.78% 7.94GiB / 23.43GiB 33.90% 1.9kB / 1.04kB 29.3GB / 11.2GB 10
As can be seen CPU usage remains high despite no vectorization or database upserts.
The docker stats outout settles down after 10 minutes to below:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
e684bb73f776 weaviate-reranker-transformers-1 0.10% 324.4MiB / 23.43GiB 1.35% 5.06kB / 5.36kB 6.83GB / 16.4kB 12
a8d91fc44be3 weaviate-multi2vec-bind-1 0.09% 5.006GiB / 23.43GiB 21.37% 1.24GB / 414MB 18.8GB / 45.1kB 40
4df61d029961 weaviate-weaviate-1 0.25% 1015MiB / 23.43GiB 4.23% 1.97kB / 1.04kB 29.4GB / 11.4GB 10
Complete weaviate logs
weaviate-weaviate-1 docker image logs
weaviate-weaviate-1 | {"action":"startup","default_vectorizer_module":"multi2vec-bind","level":"info","msg":"the default vectorizer modules is set to \"multi2vec-bind\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-01-13T12:11:35Z"}
weaviate-weaviate-1 | {"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-01-13T12:11:35Z"}
weaviate-weaviate-1 | {"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-01-13T12:11:35Z"}
weaviate-weaviate-1 | {"time":"2024-01-13T12:11:35.933325715Z","level":"INFO","msg":"open cluster service","servers":["172.18.0.2:8300"]}
weaviate-weaviate-1 | {"time":"2024-01-13T12:11:35.933375514Z","level":"INFO","msg":"starting cloud rpc server ...","address":"172.18.0.2:8301"}
weaviate-weaviate-1 | {"time":"2024-01-13T12:11:35.933445233Z","level":"INFO","msg":"bootstrapping started"}
weaviate-weaviate-1 | {"time":"2024-01-13T12:11:35.951883206Z","level":"INFO","msg":"raft tcp transport","address":"172.18.0.2:8300","tcpMaxPool":3,"tcpTimeout":10000000000}
weaviate-weaviate-1 | {"time":"2024-01-13T12:11:35.952085844Z","level":"INFO","msg":"database has been successfully loaded"}
weaviate-weaviate-1 | 2024-01-13T12:11:35.954Z [INFO] raft: initial configuration: index=0 servers=[]
weaviate-weaviate-1 | {"time":"2024-01-13T12:11:35.954164138Z","level":"INFO","msg":"starting raft","applied_index":0,"last_index":0,"last_log_index":0}
weaviate-weaviate-1 | 2024-01-13T12:11:35.954Z [INFO] raft: entering follower state: follower="Node at 172.18.0.2:8300 [Follower]" leader-address= leader-id=
weaviate-weaviate-1 | 2024-01-13T12:11:36.957Z [WARN] raft: no known peers, aborting election
weaviate-weaviate-1 | {"time":"2024-01-13T12:11:37.081518974Z","level":"INFO","msg":"starting bootstrapping","candidates":[{"Suffrage":0,"ID":"node1","Address":"172.18.0.2:8300"}]}
weaviate-weaviate-1 | 2024-01-13T12:11:38.692Z [WARN] raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
weaviate-weaviate-1 | 2024-01-13T12:11:38.692Z [INFO] raft: entering candidate state: node="Node at 172.18.0.2:8300 [Candidate]" term=2
weaviate-weaviate-1 | 2024-01-13T12:11:38.701Z [INFO] raft: election won: term=2 tally=1
weaviate-weaviate-1 | 2024-01-13T12:11:38.701Z [INFO] raft: entering leader state: leader="Node at 172.18.0.2:8300 [Leader]"
weaviate-weaviate-1 | {"time":"2024-01-13T12:11:39.326423777Z","level":"INFO","msg":"successfully joined cluster","leader":"172.18.0.2:8300"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:11:50Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:11:51Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:11:52Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:11:53Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:11:54Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:11:55Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:11:56Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:11:57Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:11:58Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:11:59Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:12:00Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:12:01Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:12:02Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:12:03Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:12:04Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:12:05Z"}
weaviate-weaviate-1 | {"time":"2024-01-13T12:12:05.955085422Z","level":"INFO","msg":"current Leader","address":"172.18.0.2:8300"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:12:06Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:12:07Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:12:08Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:12:09Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:12:10Z"}
weaviate-weaviate-1 | {"action":"multi2vec_remote_wait_for_startup","error":"send check ready request: Get \"http://multi2vec-bind:8080/.well-known/ready\": dial tcp 172.18.0.4:8080: connect: connection refused","level":"warning","msg":"multi2vec-bind inference service not ready","time":"2024-01-13T12:12:11Z"}
weaviate-weaviate-1 | {"level":"warning","msg":"Multiple vector spaces are present, GraphQL Explore and REST API list objects endpoint module include params has been disabled as a result.","time":"2024-01-13T12:12:13Z"}
weaviate-weaviate-1 | {"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-01-13T12:12:13Z"}
weaviate-weaviate-1 | {"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-01-13T12:12:13Z"}
weaviate-weaviate-1 | {"level":"info","msg":"WARNING: prop len tracker file /var/lib/weaviate/foods/fBciQ6dMkfYM/proplengths does not exist, creating new tracker","time":"2024-01-13T13:29:00Z"}
weaviate-weaviate-1 | {"level":"info","msg":"Created shard foods_fBciQ6dMkfYM in 1.093036ms","time":"2024-01-13T13:29:00Z"}
weaviate-weaviate-1 | {"action":"hnsw_vector_cache_prefill","count":1000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-01-13T13:29:00Z","took":111280}
weaviate-weaviate-1 exited with code 137
weaviate-weaviate-1 | {"action":"startup","default_vectorizer_module":"multi2vec-bind","level":"info","msg":"the default vectorizer modules is set to \"multi2vec-bind\", as a result all new schema classes without an explicit vectorizer setting, will use this vectorizer","time":"2024-01-14T05:04:33Z"}
weaviate-weaviate-1 | {"action":"startup","auto_schema_enabled":true,"level":"info","msg":"auto schema enabled setting is set to \"true\"","time":"2024-01-14T05:04:33Z"}
weaviate-weaviate-1 | {"level":"info","msg":"No resource limits set, weaviate will use all available memory and CPU. To limit resources, set LIMIT_RESOURCES=true","time":"2024-01-14T05:04:33Z"}
weaviate-weaviate-1 | {"time":"2024-01-14T05:04:33.442585409Z","level":"INFO","msg":"open cluster service","servers":["172.18.0.2:8300"]}
weaviate-weaviate-1 | {"time":"2024-01-14T05:04:33.444396322Z","level":"INFO","msg":"starting cloud rpc server ...","address":"172.18.0.2:8301"}
weaviate-weaviate-1 | {"time":"2024-01-14T05:04:33.444622641Z","level":"INFO","msg":"bootstrapping started"}
weaviate-weaviate-1 | {"time":"2024-01-14T05:04:33.451273255Z","level":"INFO","msg":"raft tcp transport","address":"172.18.0.2:8300","tcpMaxPool":3,"tcpTimeout":10000000000}
weaviate-weaviate-1 | 2024-01-14T05:04:33.460Z [INFO] raft: initial configuration: index=1 servers="[{Suffrage:Voter ID:node1 Address:172.18.0.2:8300}]"
weaviate-weaviate-1 | {"time":"2024-01-14T05:04:33.461411136Z","level":"INFO","msg":"starting raft","applied_index":0,"last_index":7,"last_log_index":7}
weaviate-weaviate-1 | 2024-01-14T05:04:33.462Z [INFO] raft: entering follower state: follower="Node at 172.18.0.2:8300 [Follower]" leader-address= leader-id=
weaviate-weaviate-1 | 2024-01-14T05:04:34.554Z [WARN] raft: heartbeat timeout reached, starting election: last-leader-addr= last-leader-id=
weaviate-weaviate-1 | 2024-01-14T05:04:34.555Z [INFO] raft: entering candidate state: node="Node at 172.18.0.2:8300 [Candidate]" term=3
weaviate-weaviate-1 | 2024-01-14T05:04:34.559Z [INFO] raft: election won: term=3 tally=1
weaviate-weaviate-1 | 2024-01-14T05:04:34.559Z [INFO] raft: entering leader state: leader="Node at 172.18.0.2:8300 [Leader]"
weaviate-weaviate-1 | {"time":"2024-01-14T05:04:34.574419566Z","level":"INFO","msg":"database has been successfully loaded"}
weaviate-weaviate-1 | {"time":"2024-01-14T05:04:35.281394335Z","level":"INFO","msg":"successfully joined cluster","leader":"172.18.0.2:8300"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal","class":"Foods","index":"foods","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/objects/segment-1705208261648586531.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal_success","class":"Foods","index":"foods","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/objects/segment-1705208261648586531.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal","class":"Foods","index":"foods","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_mediaType/segment-1705208263051126542.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal_success","class":"Foods","index":"foods","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_mediaType/segment-1705208263051126542.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal","class":"Foods","index":"foods","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_path/segment-1705208262225964359.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal_success","class":"Foods","index":"foods","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_path/segment-1705208262225964359.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal","class":"Foods","index":"foods","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_mediaType_searchable/segment-1705208263068485994.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal_success","class":"Foods","index":"foods","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_mediaType_searchable/segment-1705208263068485994.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal","class":"Foods","index":"foods","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_name/segment-1705208263076008725.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal_success","class":"Foods","index":"foods","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_name/segment-1705208263076008725.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal","class":"Foods","index":"foods","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property__id/segment-1705208262088150977.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal_success","class":"Foods","index":"foods","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property__id/segment-1705208262088150977.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal","class":"Foods","index":"foods","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_path_searchable/segment-1705208262509244455.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal_success","class":"Foods","index":"foods","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_path_searchable/segment-1705208262509244455.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal","class":"Foods","index":"foods","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_name_searchable/segment-1705208263128449000.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal_success","class":"Foods","index":"foods","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_name_searchable/segment-1705208263128449000.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:35Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal","class":"Foods","index":"foods","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_image/segment-1705208262590156979.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:36Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal_success","class":"Foods","index":"foods","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_image/segment-1705208262590156979.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:36Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal","class":"Foods","index":"foods","level":"warning","msg":"active write-ahead-log found. Did weaviate crash prior to this? Trying to recover...","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_image_searchable/segment-1705208262700560509.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:36Z"}
weaviate-weaviate-1 | {"action":"lsm_recover_from_active_wal_success","class":"Foods","index":"foods","level":"info","msg":"successfully recovered from write-ahead-log","path":"/var/lib/weaviate/foods/fBciQ6dMkfYM/lsm/property_image_searchable/segment-1705208262700560509.wal","shard":"fBciQ6dMkfYM","time":"2024-01-14T05:04:36Z"}
weaviate-weaviate-1 | {"level":"info","msg":"Completed loading shard foods_fBciQ6dMkfYM in 1.049590717s","time":"2024-01-14T05:04:36Z"}
weaviate-weaviate-1 | {"level":"warning","msg":"Multiple vector spaces are present, GraphQL Explore and REST API list objects endpoint module include params has been disabled as a result.","time":"2024-01-14T05:04:47Z"}
weaviate-weaviate-1 | {"action":"grpc_startup","level":"info","msg":"grpc server listening at [::]:50051","time":"2024-01-14T05:04:47Z"}
weaviate-weaviate-1 | {"action":"restapi_management","level":"info","msg":"Serving weaviate at http://[::]:8080","time":"2024-01-14T05:04:47Z"}
weaviate-weaviate-1 | {"action":"hnsw_vector_cache_prefill","count":23971,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-01-14T05:04:50Z","took":13697782231}
weaviate-weaviate-1 | {"time":"2024-01-14T05:05:03.462076305Z","level":"INFO","msg":"current Leader","address":"172.18.0.2:8300"}
weaviate-weaviate-1 | {"level":"info","msg":"WARNING: prop len tracker file /var/lib/weaviate/foods/BRZ1cZlA51pK/proplengths does not exist, creating new tracker","time":"2024-01-14T05:22:41Z"}
weaviate-weaviate-1 | {"action":"hnsw_vector_cache_prefill","count":1000,"index_id":"main","level":"info","limit":1000000000000,"msg":"prefilled vector cache","time":"2024-01-14T05:22:41Z","took":106080}
weaviate-weaviate-1 | {"level":"info","msg":"Created shard foods_BRZ1cZlA51pK in 3.542946ms","time":"2024-01-14T05:22:41Z"}
weaviate-multi2vec-bind-1 container log head and tail
head
weaviate-multi2vec-bind-1 | /usr/local/lib/python3.11/site-packages/torchvision/transforms/functional_tensor.py:5: UserWarning: The torchvision.transforms.functional_tensor module is deprecated in 0.15 and will be **removed in 0.17**. Please don't rely on it. You probably just need to use APIs in torchvision.transforms.functional or in torchvision.transforms.v2.functional.
weaviate-multi2vec-bind-1 | warnings.warn(
weaviate-multi2vec-bind-1 | /usr/local/lib/python3.11/site-packages/torchvision/transforms/_functional_video.py:6: UserWarning: The 'torchvision.transforms._functional_video' module is deprecated since 0.12 and will be removed in the future. Please use the 'torchvision.transforms.functional' module instead.
weaviate-multi2vec-bind-1 | warnings.warn(
weaviate-multi2vec-bind-1 | /usr/local/lib/python3.11/site-packages/torchvision/transforms/_transforms_video.py:22: UserWarning: The 'torchvision.transforms._transforms_video' module is deprecated since 0.12 and will be removed in the future. Please use the 'torchvision.transforms' module instead.
weaviate-multi2vec-bind-1 | warnings.warn(
weaviate-multi2vec-bind-1 | INFO: Started server process [7]
weaviate-multi2vec-bind-1 | INFO: Waiting for application startup.
weaviate-multi2vec-bind-1 | INFO: Running on CPU
weaviate-multi2vec-bind-1 | INFO: Model initialization complete
weaviate-multi2vec-bind-1 | INFO: Application startup complete.
weaviate-multi2vec-bind-1 | INFO: Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
weaviate-multi2vec-bind-1 | INFO: 172.18.0.2:60704 - "GET /.well-known/ready HTTP/1.1" 204 No Content
weaviate-multi2vec-bind-1 | INFO: 172.18.0.2:55990 - "GET /meta HTTP/1.1" 200 OK
weaviate-multi2vec-bind-1 | INFO: 172.18.0.2:55990 - "GET /meta HTTP/1.1" 200 OK
weaviate-multi2vec-bind-1 | INFO: 172.18.0.2:53308 - "POST /vectorize HTTP/1.1" 200 OK
tail
weaviate-multi2vec-bind-1 | INFO: 172.18.0.2:56344 - "POST /vectorize HTTP/1.1" 200 OK
weaviate-multi2vec-bind-1 | INFO: 172.18.0.2:34218 - "POST /vectorize HTTP/1.1" 200 OK
weaviate-multi2vec-bind-1 | INFO: 172.18.0.2:56344 - "POST /vectorize HTTP/1.1" 200 OK
weaviate-multi2vec-bind-1 | INFO: 172.18.0.2:32902 - "POST /vectorize HTTP/1.1" 200 OK
weaviate-multi2vec-bind-1 | INFO: 172.18.0.2:34220 - "POST /vectorize HTTP/1.1" 200 OK
weaviate-multi2vec-bind-1 | INFO: 172.18.0.2:34236 - "POST /vectorize HTTP/1.1" 200 OK
weaviate-multi2vec-bind-1 | INFO: 172.18.0.2:53218 - "POST /vectorize HTTP/1.1" 200 OK
weaviate-multi2vec-bind-1 | INFO: 172.18.0.2:35270 - "GET /.well-known/ready HTTP/1.1" 204 No Content
Unable to replicate collection delete failure
from weaviate.
Related Issues (20)
- Panic in HNSW cleanup can lead to deadlock
- [Proposal] Switch on optional indexing options by default
- High MTTR with many tenants HOT 1
- [v1.24] Tx can get stuck on some nodes when tx commit fails
- Better tenant observability
- `with_near_vector` function does not work HOT 2
- `X-Azure-Api-Key` doesn't work in headers. Asking openai api key instead. while calling withNearText search. HOT 4
- Windows is not supported with Embedded HOT 1
- Multivector: knn search: distance between entrypoint and query node: got a nil or zero-length vector at docID 1115 HOT 5
- Duplicate and inconsistent results of BM25 search
- Major loading speed regression with v1.24.9 HOT 6
- Improvements to geo-coordinate based filtering HOT 5
- explainScore containing only one score when performing hybrid search HOT 2
- Weaviate backup using S3 closed pipe http 499 after 1m 3s
- Update by Query
- Duplicate CONTEXTIONARY_URL in dev setup HOT 1
- Data encryption in weaviate for securing sensitive information
- text2vec-ollama is not working HOT 2
- Observability metrics for async indexing queue
- Existing replication factor increase implementation does not work with raft
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from weaviate.