lsst-sqre / strimzi-registry-operator Goto Github PK
View Code? Open in Web Editor NEWA Kubernetes Operator for running the Confluent Schema Registry with a Strimzi-based Kafka cluster
License: MIT License
A Kubernetes Operator for running the Confluent Schema Registry with a Strimzi-based Kafka cluster
License: MIT License
Hello, i'm trying to get the operator working on a GCP cluster with a working strimzi-kafka stack. The strimzi-kafka stack is deployed to a namespace called "kafka". The strimzi-registry-operator deployment was also done into this namespace.
A double check of the serviceaccount + role + rolebinding on the cluster seems good too. But the operator is crashing with following log entries:
[2022-03-15 08:41:12,267] kopf.clients.auth [DEBUG ] Pykube is configured in cluster with service account.
[2022-03-15 08:41:12,285] kopf.clients.auth [DEBUG ] Client is configured in cluster with service account.
[2022-03-15 08:41:12,357] kopf.reactor.running [ERROR ] Root task 'watcher of secrets.' is failed: HTTPError('403 Client Error: Forbidden for url: https://10.0.128.1:443/api/v1/namespaces/events/secrets')
[2022-03-15 08:41:12,357] kopf.reactor.running [ERROR ] Root task 'watcher of strimzischemaregistries.roundtable.lsst.codes' is failed: HTTPError('403 Client Error: Forbidden for url: https://10.0.128.1:443/apis/roundtable.lsst.codes/v1beta1/namespaces/events/strimzischemaregistries')
[2022-03-15 08:41:12,357] kopf.reactor.running [DEBUG ] Root task 'poster of events' is cancelled.
[2022-03-15 08:41:12,357] kopf.reactor.running [DEBUG ] Root tasks are stopped: finished normally; tasks left: set()
[2022-03-15 08:41:12,358] kopf.reactor.running [DEBUG ] Hung tasks stopping is skipped: no tasks given.
Traceback (most recent call last):
File "/opt/venv/bin/kopf", line 8, in <module>
Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'a33c5c91-5d93-46d6-ac7d-5496a87dfa59', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '474e4962-5d35-4d9c-9f49-981defbbe192', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'c36a8cd5-1ce3-464c-a153-99b73e582954', 'Date': 'Tue, 15 Mar 2022 08:41:12 GMT', 'Content-Length': '429'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"strimzischemaregistries.roundtable.lsst.codes is forbidden: User \"system:serviceaccount:kafka:strimzi-registry-operator\" cannot list resource \"strimzischemaregistries\" in API group \"roundtable.lsst.codes\" in the namespace \"events\"","reason":"Forbidden","details":{"group":"roundtable.lsst.codes","kind":"strimzischemaregistries"},"code":403}
sys.exit(main())
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/kopf/cli.py", line 30, in wrapper
return fn(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/kopf/cli.py", line 59, in run
return running.run(
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 38, in run
loop.run_until_complete(operator(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 79, in operator
await run_tasks(operator_tasks, ignored=existing_tasks)
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 215, in run_tasks
await _reraise(root_done | root_cancelled | hung_done | hung_cancelled)
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 263, in _reraise
task.result() # can raise the regular (non-cancellation) exceptions.
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 270, in _root_task_checker
await coro
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/queueing.py", line 80, in watcher
async for event in watching.infinite_watch(resource=resource, namespace=namespace):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 86, in infinite_watch
async for event in streaming_watch(resource=resource, namespace=namespace):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 101, in streaming_watch
rsp = fetching.list_objs(resource=resource, namespace=namespace)
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/fetching.py", line 61, in list_objs
return lst.response
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 199, in response
return self.query_cache["response"]
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 185, in query_cache
cache["response"] = self.execute().json()
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 160, in execute
r.raise_for_status()
File "/opt/venv/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.0.128.1:443/apis/roundtable.lsst.codes/v1beta1/namespaces/events/strimzischemaregistries
Task exception was never retrieved
future: <Task finished name='Task-4' coro=<_root_task_checker() done, defined at /opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py:268> exception=HTTPError('403 Client Error: Forbidden for url: https://10.0.128.1:443/api/v1/namespaces/events/secrets')>
Traceback (most recent call last):
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 270, in _root_task_checker
await coro
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/queueing.py", line 80, in watcher
async for event in watching.infinite_watch(resource=resource, namespace=namespace):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 86, in infinite_watch
async for event in streaming_watch(resource=resource, namespace=namespace):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 101, in streaming_watch
rsp = fetching.list_objs(resource=resource, namespace=namespace)
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/fetching.py", line 61, in list_objs
return lst.response
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 199, in response
return self.query_cache["response"]
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 185, in query_cache
cache["response"] = self.execute().json()
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 160, in execute
r.raise_for_status()
File "/opt/venv/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.0.128.1:443/api/v1/namespaces/events/secrets
[2022-03-15 08:42:36,404] kopf.clients.auth [DEBUG ] Pykube is configured in cluster with service account.
[2022-03-15 08:42:36,430] kopf.clients.auth [DEBUG ] Client is configured in cluster with service account.
[2022-03-15 08:42:36,515] kopf.reactor.running [ERROR ] Root task 'watcher of secrets.' is failed: HTTPError('403 Client Error: Forbidden for url: https://10.0.128.1:443/api/v1/namespaces/events/secrets')
[2022-03-15 08:42:36,515] kopf.reactor.running [ERROR ] Root task 'watcher of strimzischemaregistries.roundtable.lsst.codes' is failed: HTTPError('403 Client Error: Forbidden for url: https://10.0.128.1:443/apis/roundtable.lsst.codes/v1beta1/namespaces/events/strimzischemaregistries')
[2022-03-15 08:42:36,515] kopf.reactor.running [DEBUG ] Root task 'poster of events' is cancelled.
[2022-03-15 08:42:36,515] kopf.reactor.running [DEBUG ] Root tasks are stopped: finished normally; tasks left: set()
[2022-03-15 08:42:36,516] kopf.reactor.running [DEBUG ] Hung tasks stopping is skipped: no tasks given.
Traceback (most recent call last):
File "/opt/venv/bin/kopf", line 8, in <module>
sys.exit(main())
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/kopf/cli.py", line 30, in wrapper
return fn(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/kopf/cli.py", line 59, in run
return running.run(
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 38, in run
loop.run_until_complete(operator(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 79, in operator
await run_tasks(operator_tasks, ignored=existing_tasks)
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 215, in run_tasks
await _reraise(root_done | root_cancelled | hung_done | hung_cancelled)
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 263, in _reraise
task.result() # can raise the regular (non-cancellation) exceptions.
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 270, in _root_task_checker
await coro
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/queueing.py", line 80, in watcher
async for event in watching.infinite_watch(resource=resource, namespace=namespace):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 86, in infinite_watch
async for event in streaming_watch(resource=resource, namespace=namespace):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 101, in streaming_watch
rsp = fetching.list_objs(resource=resource, namespace=namespace)
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/fetching.py", line 61, in list_objs
return lst.response
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 199, in response
return self.query_cache["response"]
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 185, in query_cache
cache["response"] = self.execute().json()
Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '16019b00-e514-4b0a-9e5e-b69df742ffc0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '474e4962-5d35-4d9c-9f49-981defbbe192', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'c36a8cd5-1ce3-464c-a153-99b73e582954', 'Date': 'Tue, 15 Mar 2022 08:42:36 GMT', 'Content-Length': '429'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"strimzischemaregistries.roundtable.lsst.codes is forbidden: User \"system:serviceaccount:kafka:strimzi-registry-operator\" cannot list resource \"strimzischemaregistries\" in API group \"roundtable.lsst.codes\" in the namespace \"events\"","reason":"Forbidden","details":{"group":"roundtable.lsst.codes","kind":"strimzischemaregistries"},"code":403}
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 160, in execute
r.raise_for_status()
File "/opt/venv/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.0.128.1:443/apis/roundtable.lsst.codes/v1beta1/namespaces/events/strimzischemaregistries
Task exception was never retrieved
future: <Task finished name='Task-4' coro=<_root_task_checker() done, defined at /opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py:268> exception=HTTPError('403 Client Error: Forbidden for url: https://10.0.128.1:443/api/v1/namespaces/events/secrets')>
Traceback (most recent call last):
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 270, in _root_task_checker
await coro
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/queueing.py", line 80, in watcher
async for event in watching.infinite_watch(resource=resource, namespace=namespace):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 86, in infinite_watch
async for event in streaming_watch(resource=resource, namespace=namespace):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 101, in streaming_watch
rsp = fetching.list_objs(resource=resource, namespace=namespace)
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/fetching.py", line 61, in list_objs
return lst.response
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 199, in response
return self.query_cache["response"]
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 185, in query_cache
cache["response"] = self.execute().json()
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 160, in execute
r.raise_for_status()
File "/opt/venv/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.0.128.1:443/api/v1/namespaces/events/secrets
[2022-03-15 08:45:20,372] kopf.clients.auth [DEBUG ] Pykube is configured in cluster with service account.
[2022-03-15 08:45:20,398] kopf.clients.auth [DEBUG ] Client is configured in cluster with service account.
[2022-03-15 08:45:20,499] kopf.reactor.running [ERROR ] Root task 'watcher of strimzischemaregistries.roundtable.lsst.codes' is failed: HTTPError('403 Client Error: Forbidden for url: https://10.0.128.1:443/apis/roundtable.lsst.codes/v1beta1/namespaces/events/strimzischemaregistries')
[2022-03-15 08:45:20,500] kopf.reactor.running [ERROR ] Root task 'watcher of secrets.' is failed: HTTPError('403 Client Error: Forbidden for url: https://10.0.128.1:443/api/v1/namespaces/events/secrets')
[2022-03-15 08:45:20,502] kopf.reactor.running [DEBUG ] Root task 'poster of events' is cancelled.
[2022-03-15 08:45:20,503] kopf.reactor.running [DEBUG ] Root tasks are stopped: finished normally; tasks left: set()
[2022-03-15 08:45:20,503] kopf.reactor.running [DEBUG ] Hung tasks stopping is skipped: no tasks given.
Traceback (most recent call last):
File "/opt/venv/bin/kopf", line 8, in <module>
sys.exit(main())
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '66a9ffc1-87b5-420f-be78-762e90f71b1e', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '474e4962-5d35-4d9c-9f49-981defbbe192', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'c36a8cd5-1ce3-464c-a153-99b73e582954', 'Date': 'Tue, 15 Mar 2022 08:45:20 GMT', 'Content-Length': '429'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"strimzischemaregistries.roundtable.lsst.codes is forbidden: User \"system:serviceaccount:kafka:strimzi-registry-operator\" cannot list resource \"strimzischemaregistries\" in API group \"roundtable.lsst.codes\" in the namespace \"events\"","reason":"Forbidden","details":{"group":"roundtable.lsst.codes","kind":"strimzischemaregistries"},"code":403}
2022-03-15T08:45:20.504821534Z
[2022-03-15 08:50:30,195] kopf.clients.auth [DEBUG ] Pykube is configured in cluster with service account.
[2022-03-15 08:50:30,222] kopf.clients.auth [DEBUG ] Client is configured in cluster with service account.
[2022-03-15 08:50:30,300] kopf.reactor.running [ERROR ] Root task 'watcher of secrets.' is failed: HTTPError('403 Client Error: Forbidden for url: https://10.0.128.1:443/api/v1/namespaces/events/secrets')
[2022-03-15 08:50:30,300] kopf.reactor.running [ERROR ] Root task 'watcher of strimzischemaregistries.roundtable.lsst.codes' is failed: HTTPError('403 Client Error: Forbidden for url: https://10.0.128.1:443/apis/roundtable.lsst.codes/v1beta1/namespaces/events/strimzischemaregistries')
[2022-03-15 08:50:30,301] kopf.reactor.running [DEBUG ] Root task 'poster of events' is cancelled.
[2022-03-15 08:50:30,301] kopf.reactor.running [DEBUG ] Root tasks are stopped: finished normally; tasks left: set()
[2022-03-15 08:50:30,301] kopf.reactor.running [DEBUG ] Hung tasks stopping is skipped: no tasks given.
Traceback (most recent call last):
File "/opt/venv/bin/kopf", line 8, in <module>
Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ebeff016-e820-4ff0-b95a-c4265d4af1ed', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '474e4962-5d35-4d9c-9f49-981defbbe192', 'X-Kubernetes-Pf-Prioritylevel-Uid': 'c36a8cd5-1ce3-464c-a153-99b73e582954', 'Date': 'Tue, 15 Mar 2022 08:50:30 GMT', 'Content-Length': '429'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"strimzischemaregistries.roundtable.lsst.codes is forbidden: User \"system:serviceaccount:kafka:strimzi-registry-operator\" cannot list resource \"strimzischemaregistries\" in API group \"roundtable.lsst.codes\" in the namespace \"events\"","reason":"Forbidden","details":{"group":"roundtable.lsst.codes","kind":"strimzischemaregistries"},"code":403}
sys.exit(main())
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/kopf/cli.py", line 30, in wrapper
return fn(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/kopf/cli.py", line 59, in run
return running.run(
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 38, in run
loop.run_until_complete(operator(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 79, in operator
await run_tasks(operator_tasks, ignored=existing_tasks)
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 215, in run_tasks
await _reraise(root_done | root_cancelled | hung_done | hung_cancelled)
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 263, in _reraise
task.result() # can raise the regular (non-cancellation) exceptions.
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 270, in _root_task_checker
await coro
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/queueing.py", line 80, in watcher
async for event in watching.infinite_watch(resource=resource, namespace=namespace):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 86, in infinite_watch
async for event in streaming_watch(resource=resource, namespace=namespace):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 101, in streaming_watch
rsp = fetching.list_objs(resource=resource, namespace=namespace)
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/fetching.py", line 61, in list_objs
return lst.response
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 199, in response
return self.query_cache["response"]
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 185, in query_cache
cache["response"] = self.execute().json()
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 160, in execute
r.raise_for_status()
File "/opt/venv/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://10.0.128.1:443/apis/roundtable.lsst.codes/v1beta1/namespaces/events/strimzischemaregistries
This is the deployed serviceaccount:
apiVersion: v1
kind: ServiceAccount
metadata:
name: strimzi-registry-operator
namespace: kafka
uid: 2243433a-b873-4a1f-b5b0-6da27684b9f3
resourceVersion: '55436239'
creationTimestamp: '2022-03-15T08:38:54Z'
labels:
app.kubernetes.io/managed-by: Helm
annotations:
meta.helm.sh/release-name: lis-kafka-schema-registry-operator-test
meta.helm.sh/release-namespace: kafka
managedFields:
- manager: helm
operation: Update
apiVersion: v1
time: '2022-03-15T08:38:54Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:meta.helm.sh/release-name: {}
f:meta.helm.sh/release-namespace: {}
f:labels:
.: {}
f:app.kubernetes.io/managed-by: {}
- manager: kube-controller-manager
operation: Update
apiVersion: v1
time: '2022-03-15T08:38:54Z'
fieldsType: FieldsV1
fieldsV1:
f:secrets:
.: {}
k:{"name":"strimzi-registry-operator-token-m9wm9"}:
.: {}
f:name: {}
selfLink: /api/v1/namespaces/kafka/serviceaccounts/strimzi-registry-operator
secrets:
- name: strimzi-registry-operator-token-m9wm9
This is the deployed role:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: strimzi-registry-operator
namespace: kafka
uid: 452026eb-6fef-436e-ae67-ba8deb7f1115
resourceVersion: '55436243'
creationTimestamp: '2022-03-15T08:38:54Z'
labels:
app.kubernetes.io/managed-by: Helm
annotations:
meta.helm.sh/release-name: lis-kafka-schema-registry-operator-test
meta.helm.sh/release-namespace: kafka
managedFields:
- manager: helm
operation: Update
apiVersion: rbac.authorization.k8s.io/v1
time: '2022-03-15T08:38:54Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:meta.helm.sh/release-name: {}
f:meta.helm.sh/release-namespace: {}
f:labels:
.: {}
f:app.kubernetes.io/managed-by: {}
f:rules: {}
selfLink: >-
/apis/rbac.authorization.k8s.io/v1/namespaces/kafka/roles/strimzi-registry-operator
rules:
- verbs:
- list
- get
apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
- verbs:
- create
apiGroups:
- events.k8s.io
resources:
- events
- verbs:
- create
apiGroups:
- ''
resources:
- events
- verbs:
- get
- list
- watch
- patch
apiGroups:
- roundtable.lsst.codes
resources:
- strimzischemaregistries
- verbs:
- get
- list
- watch
- patch
- create
apiGroups:
- ''
resources:
- secrets
- configmaps
- services
- verbs:
- get
- list
- watch
- patch
- create
apiGroups:
- apps
resources:
- deployments
- verbs:
- list
- get
apiGroups:
- kafka.strimzi.io
resources:
- kafkausers
- kafkas
And this is the deployed rolebinding:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: strimzi-registry-operator
namespace: kafka
uid: 1c8608ea-4f9c-4ec0-93c2-2fc44f1793c2
resourceVersion: '55436244'
creationTimestamp: '2022-03-15T08:38:54Z'
labels:
app.kubernetes.io/managed-by: Helm
annotations:
meta.helm.sh/release-name: lis-kafka-schema-registry-operator-test
meta.helm.sh/release-namespace: kafka
managedFields:
- manager: helm
operation: Update
apiVersion: rbac.authorization.k8s.io/v1
time: '2022-03-15T08:38:54Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:meta.helm.sh/release-name: {}
f:meta.helm.sh/release-namespace: {}
f:labels:
.: {}
f:app.kubernetes.io/managed-by: {}
f:roleRef:
f:apiGroup: {}
f:kind: {}
f:name: {}
f:subjects: {}
selfLink: >-
/apis/rbac.authorization.k8s.io/v1/namespaces/kafka/rolebindings/strimzi-registry-operator
subjects:
- kind: ServiceAccount
name: strimzi-registry-operator
namespace: kafka
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: strimzi-registry-operator
This is the operator deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: strimzi-registry-operator
namespace: kafka
uid: 731f63f7-5994-43ef-ab15-314c096377eb
resourceVersion: '55444469'
generation: 1
creationTimestamp: '2022-03-15T08:38:54Z'
labels:
app.kubernetes.io/managed-by: Helm
annotations:
deployment.kubernetes.io/revision: '1'
meta.helm.sh/release-name: lis-kafka-schema-registry-operator-test
meta.helm.sh/release-namespace: kafka
managedFields:
- manager: helm
operation: Update
apiVersion: apps/v1
time: '2022-03-15T08:38:54Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:meta.helm.sh/release-name: {}
f:meta.helm.sh/release-namespace: {}
f:labels:
.: {}
f:app.kubernetes.io/managed-by: {}
f:spec:
f:progressDeadlineSeconds: {}
f:replicas: {}
f:revisionHistoryLimit: {}
f:selector: {}
f:strategy:
f:type: {}
f:template:
f:metadata:
f:labels:
.: {}
f:app: {}
f:spec:
f:containers:
k:{"name":"operator"}:
.: {}
f:image: {}
f:imagePullPolicy: {}
f:name: {}
f:resources: {}
f:terminationMessagePath: {}
f:terminationMessagePolicy: {}
f:dnsPolicy: {}
f:restartPolicy: {}
f:schedulerName: {}
f:securityContext: {}
f:serviceAccount: {}
f:serviceAccountName: {}
f:terminationGracePeriodSeconds: {}
- manager: kube-controller-manager
operation: Update
apiVersion: apps/v1
time: '2022-03-15T09:00:49Z'
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:deployment.kubernetes.io/revision: {}
f:status:
f:conditions:
.: {}
k:{"type":"Available"}:
.: {}
f:lastTransitionTime: {}
f:lastUpdateTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
k:{"type":"Progressing"}:
.: {}
f:lastTransitionTime: {}
f:lastUpdateTime: {}
f:message: {}
f:reason: {}
f:status: {}
f:type: {}
f:observedGeneration: {}
f:replicas: {}
f:unavailableReplicas: {}
f:updatedReplicas: {}
selfLink: /apis/apps/v1/namespaces/kafka/deployments/strimzi-registry-operator
status:
observedGeneration: 1
replicas: 1
updatedReplicas: 1
unavailableReplicas: 1
conditions:
- type: Progressing
status: 'True'
lastUpdateTime: '2022-03-15T08:39:24Z'
lastTransitionTime: '2022-03-15T08:38:55Z'
reason: NewReplicaSetAvailable
message: >-
ReplicaSet "strimzi-registry-operator-7fb5f7db9c" has successfully
progressed.
- type: Available
status: 'False'
lastUpdateTime: '2022-03-15T09:00:49Z'
lastTransitionTime: '2022-03-15T09:00:49Z'
reason: MinimumReplicasUnavailable
message: Deployment does not have minimum availability.
spec:
replicas: 1
selector:
matchLabels:
app: strimzi-registry-operator
template:
metadata:
creationTimestamp: null
labels:
app: strimzi-registry-operator
spec:
containers:
- name: operator
image: lsstsqre/strimzi-registry-operator:0.4.1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullPolicy: Always
restartPolicy: Always
terminationGracePeriodSeconds: 30
dnsPolicy: ClusterFirst
serviceAccountName: strimzi-registry-operator
serviceAccount: strimzi-registry-operator
securityContext: {}
schedulerName: default-scheduler
strategy:
type: Recreate
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
As for the Kafka Topic and User, the example from the readme was deployed after operator deployment.
Hi, I am running a local Minikube setup. Here are the steps I followed to run schema registry on k8s (Minikube).
minikube/setupkafka.sh
Minikube/deploysro.sh
Until this everything works fine.
I ran Minikube/deployregistry.sh
and it fails citing confluent-shema-registry deployment not found. failing at this step kubectl wait deployment confluent-schema-registry \ --for condition=Available=True --timeout=600s -n default
This is the error.
Error from server (NotFound): deployments.apps "confluent-schema-registry" not found
When I checked for deployments and services in default namespace I didn't find confluent-schema-registery where as secrets is created. How to debug this issue ?
kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
test-cluster-entity-operator 1/1 1 1 97m
kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 100m
test-cluster-kafka-bootstrap ClusterIP 10.96.120.130 <none> 9091/TCP,9092/TCP,9093/TCP 97m
test-cluster-kafka-brokers ClusterIP None <none> 9090/TCP,9091/TCP,9092/TCP,9093/TCP 97m
test-cluster-zookeeper-client ClusterIP 10.96.38.74 <none> 2181/TCP 98m
test-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 98m
kubectl get secrets
NAME TYPE DATA AGE
confluent-schema-registry Opaque 5 82m
test-cluster-clients-ca Opaque 1 99m
test-cluster-clients-ca-cert Opaque 3 99m
test-cluster-cluster-ca Opaque 1 99m
test-cluster-cluster-ca-cert Opaque 3 99m
test-cluster-cluster-operator-certs Opaque 4 99m
test-cluster-entity-topic-operator-certs Opaque 4 97m
test-cluster-entity-user-operator-certs Opaque 4 97m
test-cluster-kafka-brokers Opaque 4 97m
test-cluster-zookeeper-nodes Opaque 12 99m
I'm not sure we successfully enabled generic support for strimzi's v1beta1 API and different listener types after all in #11.
This is what a Kafka
listener spec looked like in a v1beta1
resource that we based the original operator implementation on:
apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
spec:
kafka:
authorization:
type: simple
listeners:
tls:
authentication:
type: tls
Then in the strimzi v1beta2
API the listener schema turned into an array:
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
spec:
kafka:
authorization:
superUsers:
- kafka-admin
type: simple
listeners:
- authentication:
type: scram-sha-512
name: plain
port: 9092
tls: false
type: internal
- authentication:
type: tls
name: tls
port: 9093
tls: true
type: internal
So our original approach at getting the listener, which was slightly updated in #11, is actually just leaning on the get
fallback:
listener_name = spec.get("listener", "plain")
kafka = k8s_cr_api.get_namespaced_custom_object(
group="kafka.strimzi.io",
version=strimzi_version,
namespace=namespace,
plural="kafkas",
name=cluster_name,
)
This is probably the reason #50 seemed like a fix there because the user had their Kafka set up with a listener using a listener named tls
โ and hence the default's tug of war between Rubin using using plain listeners, tls listeners, and our external open source users!
I think the solution we'll implement is to update how the listener name is obtained/retrieved and add an extra code path for v1beta1 that we're still using for one cluster internally.
[2022-05-10 14:18:43,288] kopf.clients.auth [DEBUG ] Pykube is configured in cluster with service account.
[2022-05-10 14:18:43,314] kopf.clients.auth [DEBUG ] Client is configured in cluster with service account.
[2022-05-10 14:18:43,454] kopf.reactor.running [ERROR ] Root task 'watcher of secrets.' is failed: HTTPError('403 Client Error: Forbidden for url: https://10.1.0.1:443/api/v1/namespaces/events/secrets')
[2022-05-10 14:18:43,454] kopf.reactor.running [ERROR ] Root task 'watcher of strimzischemaregistries.roundtable.lsst.codes' is failed: HTTPError('403 Client Error: Forbidden for url: https://10.1.0.1:443/apis/roundtable.lsst.codes/v1beta1/namespaces/events/strimzischemaregistries')
[2022-05-10 14:18:43,454] kopf.reactor.running [DEBUG ] Root task 'poster of events' is cancelled.
[2022-05-10 14:18:43,454] kopf.reactor.running [DEBUG ] Root tasks are stopped: finished normally; tasks left: set()
[2022-05-10 14:18:43,454] kopf.reactor.running [DEBUG ] Hung tasks stopping is skipped: no tasks given.
Exception when calling CustomObjectsApi->list_namespaced_custom_object: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'd4fdd21a-dda4-43c4-af0e-3eee3fdfd812', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '3bb8fad8-96d0-4f18-9dbc-72c44d43e14a', 'X-Kubernetes-Pf-Prioritylevel-Uid': '3bbd3f6a-ab95-411b-9204-c9714c6adbf4', 'Date': 'Tue, 10 May 2022 14:18:43 GMT', 'Content-Length': '433'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"strimzischemaregistries.roundtable.lsst.codes is forbidden: User \"system:serviceaccount:my-ns:strimzi-registry-operator\" cannot list resource \"strimzischemaregistries\" in API group \"roundtable.lsst.codes\" in the namespace \"events\"","reason":"Forbidden","details":{"group":"roundtable.lsst.codes","kind":"strimzischemaregistries"},"code":403}
I'm trying to deploy the operator in a dedicated namespace but, either via helm or Kustomize, the operator is installed in the default
namespace.
With helm, I tried
helm install lsstsqre/strimzi-registry-operator --name ssr --set clusterNamespace=kafka,clusterName=my-cluster
With Kustomize I tried to follow the instructions of the README and apply the patch but the operator is always deployed in the default
namespace, not in the one I want.
This was my process. I created a patch
folder in which I added kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/lsst-sqre/strimzi-registry-operator.git//manifests?ref=0.5.0
patches:
- strimzi-registry-operator-deployment.yaml
and strimzi-registry-operator-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: strimzi-registry-operator
spec:
template:
spec:
containers:
- name: operator
env:
- name: SSR_CLUSTER_NAME
value: my-cluster
- name: SSR_NAMESPACE
value: kafka
Then I apply the configuration with the command
kctl apply -k patch
Hi all,
First of all, thumbs up for this great project.
I found your project because we are using a company-supplied Strimzi Kafka cluster. This company is on a cloud journey and still getting used to technologies like IaC, ArgoCD, etc.
The new environment we set up is completely managed by ArgoCD. Because of this, we also want to manage our Topics and Schemas as code, but the latter seems to be a problem.
Do you see the need for this and/or do you have plans to support this in the near term?
-- Full disclosure: I didn't test your setup yet, because I simply didn't see a Schema CRD in the Helm Chart, so I assume it does not exist --
Thanks a lot.
I installed a Kafka cluster successfully.
I deployed the strimzi-registry-operator using #79 (comment)
I create a topic and a KafkaUser in the namespace of the kafka cluster. I then define a StrimziSchemaRegistry resource in the same namespace.
I get the following error in the oprator log:
[2023-05-09 22:04:34,392] kopf.objects [DEBUG ] [kafka/confluent-schema-registry] Patching with: {'metadata': {'annotations': {'kopf.zalando.org/touch-dummy': '2023-05-09T22:04:34.391801'}}, 'status': {'kopf': {'dummy': '2023-05-09T22:04:34.391801'}}}
[2023-05-09 22:04:34,419] kopf.objects [WARNING ] [kafka/confluent-schema-registry] Patching failed with inconsistencies: (('remove', ('status',), {'kopf': {'dummy': '2023-05-09T22:04:34.391801'}}, None),)
[2023-05-09 22:04:34,524] kopf.objects [DEBUG ] [kafka/confluent-schema-registry] Creation is in progress: {'apiVersion': 'roundtable.lsst.codes/v1beta1', 'kind': 'StrimziSchemaRegistry', 'metadata': {'annotations': {'kopf.zalando.org/create_registry': '{"started":"2023-05-09T21:05:22.061647","delayed":"2023-05-09T22:04:34.354321","purpose":"create","retries":73,"success":false,"failure":false,"message":"(404)\\nReason: Not Found\\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ea2cad20-6bd0-4489-b9e8-52ece4ae6efe\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'dc0327c8-640d-4099-9826-fd4709b6d952\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'52a754a4-bd10-4e90-ba33-04ea09f638b7\', \'Date\': \'Tue, 09 May 2023 22:03:34 GMT\', \'Content-Length\': \'224\'})\\nHTTP response body: b\'{\\"kind\\":\\"Status\\",\\"apiVersion\\":\\"v1\\",\\"metadata\\":{},\\"status\\":\\"Failure\\",\\"message\\":\\"secrets \\\\\\\\\\"confluent-schema-registry\\\\\\\\\\" not found\\",\\"reason\\":\\"NotFound\\",\\"details\\":{\\"name\\":\\"confluent-schema-registry\\",\\"kind\\":\\"secrets\\"},\\"code\\":404}\\\\n\'\\n"}', 'kopf.zalando.org/touch-dummy': '2023-05-09T22:04:34.391801', 'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"roundtable.lsst.codes/v1beta1","kind":"StrimziSchemaRegistry","metadata":{"annotations":{},"name":"confluent-schema-registry","namespace":"kafka"},"spec":{"compatibilityLevel":"forward","listener":"tls","securityProtocol":"PLAINTEXT","strimziVersion":"v1beta2"}}\n'}, 'creationTimestamp': '2023-05-09T21:05:21Z', 'generation': 5, 'managedFields': [{'apiVersion': 'roundtable.lsst.codes/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:kubectl.kubernetes.io/last-applied-configuration': {}}}, 'f:spec': {'.': {}, 'f:compatibilityLevel': {}, 'f:cpuLimit': {}, 'f:cpuRequest': {}, 'f:listener': {}, 'f:memoryLimit': {}, 'f:memoryRequest':
{}, 'f:registryImage': {}, 'f:registryImageTag': {}, 'f:securityProtocol': {}, 'f:serviceType': {}, 'f:strimzi-version': {}, 'f:strimziVersion': {}}}, 'manager': 'kubectl-client-side-apply', 'operation': 'Update', 'time': '2023-05-09T22:03:58Z'}, {'apiVersion': 'roundtable.lsst.codes/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:kopf.zalando.org/create_registry': {}, 'f:kopf.zalando.org/touch-dummy': {}}}}, 'manager': 'kopf', 'operation': 'Update', 'time': '2023-05-09T22:04:34Z'}], 'name': 'confluent-schema-registry', 'namespace': 'kafka', 'resourceVersion': '50816', 'uid': 'fd9c7949-1c48-44a7-9b69-2facc4d8df69'}, 'spec': {'compatibilityLevel': 'forward', 'cpuLimit': '', 'cpuRequest': '', 'listener': 'tls', 'memoryLimit': '', 'memoryRequest': '', 'registryImage': 'confluentinc/cp-schema-registry', 'registryImageTag': '7.2.1', 'securityProtocol': 'PLAINTEXT', 'serviceType': 'ClusterIP', 'strimzi-version': 'v1beta2', 'strimziVersion': 'v1beta2'}}
2023-05-09T22:04:34.525770275Z [2023-05-09 22:04:34,524] kopf.objects [DEBUG ] [kafka/confluent-schema-registry] Handler 'create_registry' is invoked.
[2023-05-09 22:04:34,526] kopf.objects [INFO ] [kafka/confluent-schema-registry] Creating a new Schema Registry deployment: confluent-schema-registry with listener=tls (security protocol=PLAINTEXT) and strimzi-version=v1beta2 serviceType=ClusterIP image=confluentinc/cp-schema-registry:7.2.1
[2023-05-09 22:04:34,554] kubernetes.client.re [DEBUG ] response body: {"apiVersion":"kafka.strimzi.io/v1beta2","kind":"KafkaUser","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"kafka.strimzi.io/v1beta2\",\"kind\":\"KafkaUser\",\"metadata\":{\"annotations\":{},\"labels\":{\"strimzi.io/cluster\":\"my-cluster\"},\"name\":\"confluent-schema-registry\",\"namespace\":\"kafka\"},\"spec\":{\"authentication\":{\"type\":\"tls\"},\"authorization\":{\"acls\":[{\"operation\":\"All\",\"resource\":{\"name\":\"registry-schemas\",\"patternType\":\"literal\",\"type\":\"topic\"},\"type\":\"allow\"},{\"operation\":\"All\",\"resource\":{\"name\":\"schema-registry\",\"patternType\":\"prefix\",\"type\":\"group\"},\"type\":\"allow\"},{\"operation\":\"Describe\",\"resource\":{\"name\":\"__consumer_offsets\",\"patternType\":\"literal\",\"type\":\"topic\"},\"type\":\"allow\"}],\"type\":\"simple\"}}}\n"},"creationTimestamp":"2023-05-09T21:57:01Z","generation":1,"labels":{"strimzi.io/cluster":"my-cluster"},"managedFields":[{"apiVersion":"kafka.strimzi.io/v1beta2","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:strimzi.io/cluster":{}}},"f:spec":{".":{},"f:authentication":{".":{},"f:type":{}},"f:authorization":{".":{},"f:acls":{},"f:type":{}}}},"manager":"kubectl-client-side-apply","operation":"Update","time":"2023-05-09T21:57:01Z"},{"apiVersion":"kafka.strimzi.io/v1beta2","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:conditions":{},"f:observedGeneration":{}}},"manager":"strimzi-user-operator","operation":"Update","subresource":"status","time":"2023-05-09T21:58:10Z"}],"name":"confluent-schema-registry","namespace":"kafka","resourceVersion":"50364","uid":"366e5c0d-e804-464d-bd2f-5814e49ca921"},"spec":{"authentication":{"type":"tls"},"authorization":{"acls":[{"operation":"All","resource":{"name":"registry-schemas","patternType":"literal","type":"to
pic"},"type":"allow"},{"operation":"All","resource":{"name":"schema-registry","patternType":"prefix","type":"group"},"type":"allow"},{"operation":"Describe","resource":{"name":"__consumer_offsets","patternType":"literal","type":"topic"},"type":"allow"}],"type":"simple"}},"status":{"conditions":[{"lastTransitionTime":"2023-05-09T21:58:10.138163275Z","message":"io.strimzi.operator.common.ReconciliationException: io.strimzi.operator.cluster.model.InvalidResourceException: Simple authorization ACL rules are configured but not supported in the Kafka cluster configuration.","reason":"ExecutionException","status":"True","type":"NotReady"},{"lastTransitionTime":"2023-05-09T21:58:10.046348275Z","message":"In API version kafka.strimzi.io/v1beta2 the operation property at path spec.authorization.acls.operation has been deprecated, and should now be configured using spec.authorization.acls[*].operations.","reason":"DeprecatedFields","status":"True","type":"Warning"}],"observedGeneration":1}}
[2023-05-09 22:04:34,570] kubernetes.client.re [DEBUG ] response body: {"apiVersion":"kafka.strimzi.io/v1beta2","kind":"Kafka","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"kafka.strimzi.io/v1beta2\",\"kind\":\"Kafka\",\"metadata\":{\"annotations\":{},\"labels\":{\"skaffold.dev/run-id\":\"0271e368-f57e-4882-a9e7-15d77c6a5b38\"},\"name\":\"my-cluster\",\"namespace\":\"kafka\"},\"spec\":{\"entityOperator\":{\"topicOperator\":{},\"userOperator\":{}},\"kafka\":{\"config\":{\"default.replication.factor\":1,\"inter.broker.protocol.version\":\"3.4\",\"min.insync.replicas\":1,\"offsets.topic.replication.factor\":1,\"transaction.state.log.min.isr\":1,\"transaction.state.log.replication.factor\":1},\"listeners\":[{\"name\":\"plain\",\"port\":9092,\"tls\":false,\"type\":\"internal\"},{\"name\":\"tls\",\"port\":9093,\"tls\":true,\"type\":\"internal\"}],\"replicas\":1,\"storage\":{\"type\":\"ephemeral\"},\"version\":\"3.4.0\"},\"zookeeper\":{\"replicas\":3,\"storage\":{\"type\":\"ephemeral\"}}}}\n"},"creationTimestamp":"2023-05-09T21:57:01Z","generation":1,"labels":{"skaffold.dev/run-id":"0271e368-f57e-4882-a9e7-15d77c6a5b38"},"managedFields":[{"apiVersion":"kafka.strimzi.io/v1beta2","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}},"f:labels":{".":{},"f:skaffold.dev/run-id":{}}},"f:spec":{".":{},"f:entityOperator":{".":{},"f:topicOperator":{},"f:userOperator":{}},"f:kafka":{".":{},"f:config":{".":{},"f:default.replication.factor":{},"f:inter.broker.protocol.version":{},"f:min.insync.replicas":{},"f:offsets.topic.replication.factor":{},"f:transaction.state.log.min.isr":{},"f:transaction.state.log.replication.factor":{}},"f:listeners":{},"f:replicas":{},"f:storage":{".":{},"f:type":{}},"f:version":{}},"f:zookeeper":{".":{},"f:replicas":{},"f:storage":{".":{},"f:type":{}}}}},"manager":"kubectl-client-side-apply","operation":"Update","time":"2023-05-09T21:57:01Z"},{"
apiVersion":"kafka.strimzi.io/v1beta2","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:clusterId":{},"f:conditions":{},"f:listeners":{},"f:observedGeneration":{}}},"manager":"strimzi-cluster-operator","operation":"Update","subresource":"status","time":"2023-05-09T21:58:27Z"}],"name":"my-cluster","namespace":"kafka","resourceVersion":"50398","uid":"4c6e0fe5-144a-48d8-a5df-a56f3c11d732"},"spec":{"entityOperator":{"topicOperator":{},"userOperator":{}},"kafka":{"config":{"default.replication.factor":1,"inter.broker.protocol.version":"3.4","min.insync.replicas":1,"offsets.topic.replication.factor":1,"transaction.state.log.min.isr":1,"transaction.state.log.replication.factor":1},"listeners":[{"name":"plain","port":9092,"tls":false,"type":"internal"},{"name":"tls","port":9093,"tls":true,"type":"internal"}],"replicas":1,"storage":{"type":"ephemeral"},"version":"3.4.0"},"zookeeper":{"replicas":3,"storage":{"type":"ephemeral"}}},"status":{"clusterId":"k3gEDCRaQ3C6j4IFOUK4FQ","conditions":[{"lastTransitionTime":"2023-05-09T21:57:31.682461275Z","message":"A Kafka cluster with a single replica and ephemeral storage will lose topic messages after any restart or rolling update.","reason":"KafkaStorage","status":"True","type":"Warning"},{"lastTransitionTime":"2023-05-09T21:58:27.855718275Z","status":"True","type":"Ready"}],"listeners":[{"addresses":[{"host":"my-cluster-kafka-bootstrap.kafka.svc","port":9092}],"bootstrapServers":"my-cluster-kafka-bootstrap.kafka.svc:9092","name":"plain","type":"plain"},{"addresses":[{"host":"my-cluster-kafka-bootstrap.kafka.svc","port":9093}],"bootstrapServers":"my-cluster-kafka-bootstrap.kafka.svc:9093","certificates":["-----BEGIN CERTIFICATE-----\nMIIFLTCCAxWgAwIBAgIUbY94tQI7UVourX/zrZMb0zoRDeQwDQYJKoZIhvcNAQEN\nBQAwLTETMBEGA1UECgwKaW8uc3RyaW16aTEWMBQGA1UEAwwNY2x1c3Rlci1jYSB2\nMDAeFw0yMzA1MDkyMTU3MDJaFw0yNDA1MDgyMTU3MDJaMC0xEzARBgNVBAoMCmlv\nLnN0cmltemkxFjAUBgNVBAMMDWNsdXN0ZXItY2EgdjAwggIiMA0GCSqGSIb3DQEB\nAQUAA4ICDwAwggIKAoICAQDALy3f77g6YGeWumyylwZMTsvcwkPNRp9UoHw7fniE\nOy85vbf+K9nDg
+OyDm9g8TgY1rx9SzEKe7FMcWufUp3gPjX1yrkmRBhKd4u5nAvJ\n5uTvkRfGDAxWqp943LIDtBYik2FXkgW8KIc0JbZyr2THfp/P/48f+/QRPUopyP3s\nIZXNNGL7Rk4Suo9yooUAhy9I2U2RhLNAJPkKmKH8tHTzc9YMzkZaDI5J4n81vRTC\nq1shIvbbe2KzjD5j8kfRLg0UAIzc3/seYjiNqchRWCVG7Zc6utbz3GKUzlL2IE3n\nZp6jAEsAgySsgm/PbHIEpWLqrtOKCY5OkbrwsZc0Kez0CAG9gGwdbvRhUmF0porC\nx4/h5Rf8NBL5qXlXawc3BHgNJ4yqJmJ3eROBKOsd6GteKgirugnj3blotfT8/5Qv\nBakTbQ04kplfVxDqkyb/OdpgpkIKNKzo/zhHjkB4ODrsiKdOS2Vhtgm4jaCOUIBn\nAVtDyKmM93N/0pVKZkoVkLbJT8WERF4w2dmTgE3tZk/JDtP4OrxNcW9Th+1XT8RD\nsu/Bn5m5lD8mxE8Wl9V7Ly0ej7czsk8IiVsXInWKFPMK0VGu6MkNzgsDCiMgP5h6\ner0vVNB+IwSvXPBM3SNvREN8AV5dxKEEx95vsPxi75rv+GO6gWmJmmAtG6FoOjoZ\n6wIDAQABo0UwQzAdBgNVHQ4EFgQUxKQACD3XfQuda9b+xFqp+KjroA4wEgYDVR0T\nAQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAQYwDQYJKoZIhvcNAQENBQADggIB\nAGTY6d5dBVZylLwaryQZRQeloXPhv8NxDue9+nfn6H5FKpJ/fCa6ZiRBY7HcG5SU\nxkxWsNiHQkKApRUjdUD5KC7n4AneMl7G5Ahg5hhtwxSwANTAa+byaz8WazVBx4K7\n59ZLEUFitT5bQajUIzllDU31ai8/Yb4fh+eSgPNKODaTnLbPPZjEI1pip37F13Oo\ngciV2p9YgLqkTs7a9zt7Tp07P8f+KEs3Q8/GYrQzKaFrkp8NAqqBf64CQbbHnsGd\nZzMA964d6SwFCNzvpedyYXAkV+EAfQkAPGdwnkjxGlkem9yI84GYq2jKZjig+d9q\nHQ7rA1goHidHOKDg6qy5kPN4+92sK6GR08AfpyUIcObhGzPNgBjnKA94kGKhffiq\nm3sb8n5Z0AtcjGfxo/AIrDtt6yk2S3yfbuDMirKlsZk6t9FeKMIB+zHYKZ5PvM6v\n+0VN0SgUvZ38tDFoOjdF8BFaUtCZiiL+o/5L5bJm6Iio/RNR703Gs/pvXjbQjAtO\nH1JV4Q5WOIQgrMAv/OkCW37yUeclHC1wBiXfQDxrP4ElPISkle04We9y6F02HW6t\nIzDNf8P0kfpsEm+yVCVkjKqIt8MrKkOZROHtg/yjmJfBDfmB1lnaB+141KmZfPWk\n9dQKjsVvAENFV3O6qCsnm59qIhbchdaq6Efy3+EFHF7w\n-----END CERTIFICATE-----\n"],"name":"tls","type":"tls"}],"observedGeneration":1}}
2023-05-09T22:04:34.572555275Z
[2023-05-09 22:04:34,589] kopf.objects [INFO ] [kafka/confluent-schema-registry] Retrieved cluster CA certificate
[2023-05-09 22:04:34,592] kopf.objects [INFO ] [kafka/confluent-schema-registry] Cluster CA certificate version: 50095
[2023-05-09 22:04:34,626] kopf.objects [ERROR ] [kafka/confluent-schema-registry] Handler 'create_registry' failed with an exception. Will retry.
2023-05-09T22:04:34.628674275Z Traceback (most recent call last):
2023-05-09T22:04:34.628721275Z File "/opt/venv/lib/python3.10/site-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once
result = await invoke_handler(
2023-05-09T22:04:34.628796275Z File "/opt/venv/lib/python3.10/site-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler
2023-05-09T22:04:34.628831275Z result = await invocation.invoke(
File "/opt/venv/lib/python3.10/site-packages/kopf/_core/actions/invocation.py", line 139, in invoke
2023-05-09T22:04:34.628901275Z await asyncio.shield(future) # slightly expensive: creates tasks
2023-05-09T22:04:34.628939275Z File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/venv/lib/python3.10/site-packages/strimziregistryoperator/handlers/createregistry.py", line 136, in create_registry
secret = create_secret(
File "/opt/venv/lib/python3.10/site-packages/strimziregistryoperator/certprocessor.py", line 76, in create_secret
client_secret = get_secret(
File "/opt/venv/lib/python3.10/site-packages/strimziregistryoperator/k8s.py", line 120, in get_secret
2023-05-09T22:04:34.629193275Z result = api.read_namespaced_secret(
2023-05-09T22:04:34.629481275Z File "/opt/venv/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 24803, in read_namespaced_secret
2023-05-09T22:04:34.629596275Z return self.read_namespaced_secret_with_http_info(name, namespace, **kwargs) # noqa: E501
2023-05-09T22:04:34.629671275Z File "/opt/venv/lib/python3.10/site-packages/kubernetes/client/api/core_v1_api.py", line 24890, in read_namespaced_secret_with_http_info
2023-05-09T22:04:34.629783275Z return self.api_client.call_api(
File "/opt/venv/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 348, in call_api
2023-05-09T22:04:34.629928275Z return self.__call_api(resource_path, method,
2023-05-09T22:04:34.629979275Z File "/opt/venv/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 180, in __call_api
2023-05-09T22:04:34.630036275Z response_data = self.request(
2023-05-09T22:04:34.630150275Z File "/opt/venv/lib/python3.10/site-packages/kubernetes/client/api_client.py", line 373, in request
2023-05-09T22:04:34.630216275Z return self.rest_client.GET(url,
File "/opt/venv/lib/python3.10/site-packages/kubernetes/client/rest.py", line 240, in GET
2023-05-09T22:04:34.630293275Z return self.request("GET", url,
2023-05-09T22:04:34.630327275Z File "/opt/venv/lib/python3.10/site-packages/kubernetes/client/rest.py", line 234, in request
2023-05-09T22:04:34.630362275Z raise ApiException(http_resp=r)
2023-05-09T22:04:34.630401275Z kubernetes.client.exceptions.ApiException: (404)
2023-05-09T22:04:34.630436275Z Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'ce079e4b-5895-46d4-a64b-281554396d8f', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'dc0327c8-640d-4099-9826-fd4709b6d952', 'X-Kubernetes-Pf-Prioritylevel-Uid': '52a754a4-bd10-4e90-ba33-04ea09f638b7', 'Date': 'Tue, 09 May 2023 22:04:34 GMT', 'Content-Length': '224'})
2023-05-09T22:04:34.630508275Z HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\"confluent-schema-registry\\" not found","reason":"NotFound","details":{"name":"confluent-schema-registry","kind":"secrets"},"code":404}\n'
2023-05-09T22:04:34.630541275Z
2023-05-09T22:04:34.631328275Z [2023-05-09 22:04:34,628] kopf.objects [DEBUG ] [kafka/confluent-schema-registry] Patching with: {'metadata': {'annotations': {'kopf.zalando.org/create_registry': '{"started":"2023-05-09T21:05:22.061647","delayed":"2023-05-09T22:05:34.627706","purpose":"create","retries":74,"success":false,"failure":false,"message":"(404)\\nReason: Not Found\\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ce079e4b-5895-46d4-a64b-281554396d8f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'dc0327c8-640d-4099-9826-fd4709b6d952\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'52a754a4-bd10-4e90-ba33-04ea09f638b7\', \'Date\': \'Tue, 09 May 2023 22:04:34 GMT\', \'Content-Length\': \'224\'})\\nHTTP response body: b\'{\\"kind\\":\\"Status\\",\\"apiVersion\\":\\"v1\\",\\"metadata\\":{},\\"status\\":\\"Failure\\",\\"message\\":\\"secrets \\\\\\\\\\"confluent-schema-registry\\\\\\\\\\" not found\\",\\"reason\\":\\"NotFound\\",\\"details\\":{\\"name\\":\\"confluent-schema-registry\\",\\"kind\\":\\"secrets\\"},\\"code\\":404}\\\\n\'\\n"}', 'kopf.zalando.org/touch-dummy': None}}, 'status': {'kopf': {'progress': {'create_registry': {'started': '2023-05-09T21:05:22.061647', 'stopped': None, 'delayed': '2023-05-09
T22:05:34.627706', 'purpose': 'create', 'retries': 74, 'success': False, 'failure': False, 'message': '(404)\nReason: Not Found\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ce079e4b-5895-46d4-a64b-281554396d8f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'dc0327c8-640d-4099-9826-fd4709b6d952\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'52a754a4-bd10-4e90-ba33-04ea09f638b7\', \'Date\': \'Tue, 09 May 2023 22:04:34 GMT\', \'Content-Length\': \'224\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\\\"confluent-schema-registry\\\\" not found","reason":"NotFound","details":{"name":"confluent-schema-registry","kind":"secrets"},"code":404}\\n\'\n', 'subrefs': None}}}}}
[2023-05-09 22:04:34,666] kopf.objects [WARNING ] [kafka/confluent-schema-registry] Patching failed with inconsistencies: (('remove', ('status',), {'kopf': {'progress': {'create_registry': {'started': '2023-05-09T21:05:22.061647', 'stopped': None, 'delayed': '2023-05-09T22:05:34.627706', 'purpose': 'create', 'retries': 74, 'success': False, 'failure': False, 'message': '(404)\nReason: Not Found\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ce079e4b-5895-46d4-a64b-281554396d8f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'dc0327c8-640d-4099-9826-fd4709b6d952\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'52a754a4-bd10-4e90-ba33-04ea09f638b7\', \'Date\': \'Tue, 09 May 2023 22:04:34 GMT\', \'Content-Length\': \'224\'})\nHTTP response body: b\'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"secrets \\\\"confluent-schema-registry\\\\" not found","reason":"NotFound","details":{"name":"confluent-schema-registry","kind":"secrets"},"code":404}\\n\'\n', 'subrefs': None}}}}, None),)
2023-05-09T22:04:34.667028275Z [2023-05-09 22:04:34,666] kopf.objects [DEBUG ] [kafka/confluent-schema-registry] Sleeping was skipped because of the patch, 59.999783 seconds left.
[2023-05-09 22:04:34,770] kopf.objects [DEBUG ] [kafka/confluent-schema-registry] Creation is in progress: {'apiVersion': 'roundtable.lsst.codes/v1beta1', 'kind': 'StrimziSchemaRegistry', 'metadata': {'annotations': {'kopf.zalando.org/create_registry': '{"started":"2023-05-09T21:05:22.061647","delayed":"2023-05-09T22:05:34.627706","purpose":"create","retries":74,"success":false,"failure":false,"message":"(404)\\nReason: Not Found\\nHTTP response headers: HTTPHeaderDict({\'Audit-Id\': \'ce079e4b-5895-46d4-a64b-281554396d8f\', \'Cache-Control\': \'no-cache, private\', \'Content-Type\': \'application/json\', \'X-Kubernetes-Pf-Flowschema-Uid\': \'dc0327c8-640d-4099-9826-fd4709b6d952\', \'X-Kubernetes-Pf-Prioritylevel-Uid\': \'52a754a4-bd10-4e90-ba33-04ea09f638b7\', \'Date\': \'Tue, 09 May 2023 22:04:34 GMT\', \'Content-Length\': \'224\'})\\nHTTP response body: b\'{\\"kind\\":\\"Status\\",\\"apiVersion\\":\\"v1\\",\\"metadata\\":{},\\"status\\":\\"Failure\\",\\"message\\":\\"secrets \\\\\\\\\\"confluent-schema-registry\\\\\\\\\\" not found\\",\\"reason\\":\\"NotFound\\",\\"details\\":{\\"name\\":\\"confluent-schema-registry\\",\\"kind\\":\\"secrets\\"},\\"code\\":404}\\\\n\'\\n"}', 'kubectl.kubernetes.io/last-applied-configuration': '{"apiVersion":"roundtable.lsst.codes/v1beta1","kind":"StrimziSchemaRegistry","metadata":{"annotations":{},"name":"confluent-schema-registry","namespace":"kafka"},"spec":{"compatibilityLevel":"forward","listener":"tls","securityProtocol":"PLAINTEXT","strimziVersion":"v1beta2"}}\n'}, 'creationTimestamp': '2023-05-09T21:05:21Z', 'generation': 5, 'managedFields': [{'apiVersion': 'roundtable.lsst.codes/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'.': {}, 'f:kubectl.kubernetes.io/last-applied-configuration': {}}}, 'f:spec': {'.': {}, 'f:compatibilityLevel': {}, 'f:cpuLimit': {}, 'f:cpuRequest': {}, 'f:listener': {}, 'f:memoryLimit': {}, 'f:memoryRequest': {}, 'f:registryImage': {}, 'f:registryImageTag': {}, 'f:securi
tyProtocol': {}, 'f:serviceType': {}, 'f:strimzi-version': {}, 'f:strimziVersion': {}}}, 'manager': 'kubectl-client-side-apply', 'operation': 'Update', 'time': '2023-05-09T22:03:58Z'}, {'apiVersion': 'roundtable.lsst.codes/v1beta1', 'fieldsType': 'FieldsV1', 'fieldsV1': {'f:metadata': {'f:annotations': {'f:kopf.zalando.org/create_registry': {}}}}, 'manager': 'kopf', 'operation': 'Update', 'time': '2023-05-09T22:04:34Z'}], 'name': 'confluent-schema-registry', 'namespace': 'kafka', 'resourceVersion': '50820', 'uid': 'fd9c7949-1c48-44a7-9b69-2facc4d8df69'}, 'spec': {'compatibilityLevel': 'forward', 'cpuLimit': '', 'cpuRequest': '', 'listener': 'tls', 'memoryLimit': '', 'memoryRequest': '', 'registryImage': 'confluentinc/cp-schema-registry', 'registryImageTag': '7.2.1', 'securityProtocol': 'PLAINTEXT', 'serviceType': 'ClusterIP', 'strimzi-version': 'v1beta2', 'strimziVersion': 'v1beta2'}}
2023-05-09T22:04:34.772572275Z [2023-05-09 22:04:34,771] kopf.objects [DEBUG ] [kafka/confluent-schema-registry] Sleeping for 59.856435 seconds for the delayed handlers.
We currently hardcode the Schema Registry image version:
We should develop a pattern for updating this/maintaining this:
At the moment I believe if reading code serves me still well, there is no support for an authentication/authz on operator level where users can with username
and password
authenticate via the Basic HTTP authentication mechanism.
From what I have seen API is naked at the moment, meaning, whoever passes any request to either create or delete schema it will be executed.
Is there possibility to maybe start working on this feature? Is there maybe a proposal page to write a proposal or?
Looking forward :)
The operator fetches Strimzi custom resources under the API version v1beta1
. For example:
v1beta1
was replaced with v1beta2
in Strimzi v0.22. As explained from this April 29, 2021 blog post, the v1beta1
API was entirely removed in Strimzi v0.23.
As a result, the create_registry
handler never succeeds on Strimzi versions past 0.22; they just get a 404:
[2021-10-20 17:42:03,636] kopf.objects [INFO ] [kafka/confluent-schema-registry] Creating a new registry deployment: "confluent-schema-registry"
[2021-10-20 17:42:03,654] kopf.objects [ERROR ] [kafka/confluent-schema-registry] Handler 'create_registry' failed with an exception. Will retry.
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/handling.py", line 391, in _execute
lifecycle=lifecycle, # just a default for the sub-handlers, not used directly.
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/handling.py", line 488, in _call_handler
**kwargs,
File "/usr/local/lib/python3.7/site-packages/kopf/reactor/invocation.py", line 79, in invoke
result = await loop.run_in_executor(config.WorkersConfig.get_syn_executor(), real_fn)
File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.7/site-packages/strimziregistryoperator/handlers/createregistry.py", line 41, in create_registry
name=cluster_name
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/apis/custom_objects_api.py", line 931, in get_namespaced_custom_object
(data) = self.get_namespaced_custom_object_with_http_info(group, version, namespace, plural, name, **kwargs)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/apis/custom_objects_api.py", line 1031, in get_namespaced_custom_object_with_http_info
collection_formats=collection_formats)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 334, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 168, in __call_api
_request_timeout=_request_timeout)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/api_client.py", line 355, in request
headers=headers)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 231, in GET
query_params=query_params)
File "/usr/local/lib/python3.7/site-packages/kubernetes/client/rest.py", line 222, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'c3ae11a8-efb8-47eb-a55e-767109e1bd2a', 'Cache-Control': 'no-cache, private', 'Content-Type': 'text/plain; charset=utf-8', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '82ffa779-dd6e-4979-9073-6d6527a12a67', 'X-Kubernetes-Pf-Prioritylevel-Uid': '7ab3ebaf-3c64-43ed-accc-590d562d6d4d', 'Date': 'Wed, 20 Oct 2021 17:42:03 GMT', 'Content-Length': '19'})
HTTP response body: 404 page not found
When I try to make test
, or when I run the container built with make image
, the kopf handler immediately exits saying it can't find the async_timeout
module, a dependency of aiojobs
. I think aiojobs
is a dependency of kopf
?
Anyway, adding async_timeout==3.0.1
to install_requires
in setup.py
seems to fix this.
I am deploying this operator in AKS using tls.
However, upon starting "StrimziSchemaRegistry" I get this error in registry operator -
Traceback (most recent call last):
File "/opt/venv/lib/python3.10/site-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once
result = await invoke_handler(
File "/opt/venv/lib/python3.10/site-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler
result = await invocation.invoke(
File "/opt/venv/lib/python3.10/site-packages/kopf/_core/actions/invocation.py", line 139, in invoke
await asyncio.shield(future) # slightly expensive: creates tasks
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/venv/lib/python3.10/site-packages/strimziregistryoperator/handlers/createregistry.py", line 131, in create_registry
bootstrap_server = get_kafka_bootstrap_server(
File "/opt/venv/lib/python3.10/site-packages/strimziregistryoperator/deployments.py", line 83, in get_kafka_bootstrap_server
raise kopf.Error(msg, delay=10)
AttributeError: module 'kopf' has no attribute 'Error'
[2022-11-23 15:47:42,760] kopf.objects [DEBUG ] [tls-kafka/confluent-schema-registry] Patching with: {'metadata': {'annotations': {'kopf.zalando.org/create_registry': '{"started":"2022-11-23T08:18:50.979989","delayed":"2022-11-23T15:48:42.760410","purpose":"create","retries":438,"success":false,"failure":false,"message":"module \'kopf\' has no attribute \'Error\'"}', 'kopf.zalando.org/touch-dummy': None}}, 'status': {'kopf': {'progress': {'create_registry': {'started': '2022-11-23T08:18:50.979989', 'stopped': None, 'delayed': '2022-11-23T15:48:42.760410', 'purpose': 'create', 'retries': 438, 'success': False, 'failure': False, 'message': "module 'kopf' has no attribute 'Error'", 'subrefs': None}}}}}
[2022-11-23 15:47:42,788] kopf.objects [WARNING ] [tls-kafka/confluent-schema-registry] Patching failed with inconsistencies: (('remove', ('status',), {'kopf': {'progress': {'create_registry': {'started': '2022-11-23T08:18:50.979989', 'stopped': None, 'delayed': '2022-11-23T15:48:42.760410', 'purpose': 'create', 'retries': 438, 'success': False, 'failure': False, 'message': "module 'kopf' has no attribute 'Error'", 'subrefs': None}}}}, None),)
[2022-11-23 15:47:42,788] kopf.objects [DEBUG ] [tls-kafka/confluent-schema-registry] Sleeping was skipped because of the patch, 59.999766 seconds left.
Using this manifest for StrimziSchemaRegistry
apiVersion: roundtable.lsst.codes/v1beta1
kind: StrimziSchemaRegistry
metadata:
name: confluent-schema-registry
spec:
strimziVersion: v1beta2
listener: tls
securityProtocol: tls
compatibilityLevel: forward
registryImage: confluentinc/cp-schema-registry
registryImageTag: "7.2.2"
cpuLimit: ""
cpuRequest: ""
memoryLimit: ""
memoryRequest: ""
I am stuck now, with no idea where to debug. Any help is appreciated.
Hi Team,
We are getting below error when we are trying to apply kind: StrimziSchemaRegistry
used these environment variables in schema registry operator deployment
env:
- name: SSR_CLUSTER_NAME
value: "cdt-west-1-shared-emp-kafka"
- name: SSR_NAMESPACE
value: "emp-kafka"
[2022-05-16 14:36:03,617] kopf.objects [DEBUG ] [emp-kafka/confluent-schema-registry] Invoking handler 'handle_secret_change'.
[2022-05-16 14:36:03,617] kopf.objects [INFO ] [emp-kafka/confluent-schema-registry] Handler 'handle_secret_change' succeeded.
[2022-05-16 14:37:30,886] kopf.reactor.running [ERROR ] Root task 'watcher of strimzischemaregistries.roundtable.lsst.codes' is failed: ChunkedEncodingError(ProtocolError("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer')))
[2022-05-16 14:37:30,886] kopf.reactor.running [DEBUG ] Root task 'poster of events' is cancelled.
[2022-05-16 14:37:30,886] kopf.reactor.running [DEBUG ] Root task 'watcher of secrets.' is cancelled.
[2022-05-16 14:37:30,886] kopf.reactor.running [DEBUG ] Root tasks are stopped: finished normally; tasks left: set()
[2022-05-16 14:37:30,886] kopf.reactor.running [DEBUG ] Hung tasks stopping is skipped: no tasks given.
Traceback (most recent call last):
File "/opt/venv/lib/python3.9/site-packages/urllib3/response.py", line 438, in _error_catcher
yield
File "/opt/venv/lib/python3.9/site-packages/urllib3/response.py", line 764, in read_chunked
self._update_chunk_length()
File "/opt/venv/lib/python3.9/site-packages/urllib3/response.py", line 694, in _update_chunk_length
line = self._fp.fp.readline()
File "/usr/local/lib/python3.9/socket.py", line 704, in readinto
return self._sock.recv_into(b)
File "/usr/local/lib/python3.9/ssl.py", line 1241, in recv_into
return self.read(nbytes, buffer)
File "/usr/local/lib/python3.9/ssl.py", line 1099, in read
return self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venv/lib/python3.9/site-packages/requests/models.py", line 760, in generate
for chunk in self.raw.stream(chunk_size, decode_content=True):
File "/opt/venv/lib/python3.9/site-packages/urllib3/response.py", line 572, in stream
for line in self.read_chunked(amt, decode_content=decode_content):
File "/opt/venv/lib/python3.9/site-packages/urllib3/response.py", line 793, in read_chunked
self._original_response.close()
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/opt/venv/lib/python3.9/site-packages/urllib3/response.py", line 455, in _error_catcher
raise ProtocolError("Connection broken: %r" % e, e)
urllib3.exceptions.ProtocolError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/venv/bin/kopf", line 8, in <module>
sys.exit(main())
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/kopf/cli.py", line 30, in wrapper
return fn(*args, **kwargs)
File "/opt/venv/lib/python3.9/site-packages/kopf/cli.py", line 59, in run
return running.run(
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 38, in run
loop.run_until_complete(operator(
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 79, in operator
await run_tasks(operator_tasks, ignored=existing_tasks)
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 215, in run_tasks
await _reraise(root_done | root_cancelled | hung_done | hung_cancelled)
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 263, in _reraise
task.result() # can raise the regular (non-cancellation) exceptions.
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/running.py", line 270, in _root_task_checker
await coro
File "/opt/venv/lib/python3.9/site-packages/kopf/reactor/queueing.py", line 80, in watcher
async for event in watching.infinite_watch(resource=resource, namespace=namespace):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 86, in infinite_watch
async for event in streaming_watch(resource=resource, namespace=namespace):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 114, in streaming_watch
async for event in streaming_aiter(stream, loop=loop):
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 66, in streaming_aiter
yield await loop.run_in_executor(executor, streaming_next, src)
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 54, in streaming_next
return next(src)
File "/opt/venv/lib/python3.9/site-packages/kopf/clients/watching.py", line 159, in <genexpr>
return iter({'type': event.type, 'object': event.object.obj} for event in src)
File "/opt/venv/lib/python3.9/site-packages/pykube/query.py", line 223, in object_stream
for line in r.iter_lines():
File "/opt/venv/lib/python3.9/site-packages/requests/models.py", line 804, in iter_lines
for chunk in self.iter_content(chunk_size=chunk_size, decode_unicode=decode_unicode):
File "/opt/venv/lib/python3.9/site-packages/requests/models.py", line 763, in generate
raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
Currently, this operator only supports one kafka cluster per deployment. Strimzi supports multiple clusters deployed to various different namespaces. Is there a way for this operator to deploy multiple schema registries to each strimzi kafka cluster within a k8's cluster? Would trhere be problems if I deployed multiple instances of the operator to the k8's cluster (one per strimzi kafka cluster)?
Via #77 (comment) its clear it'd be useful for the StrimziSchemaRegistry resource to be more explicit about the Kafka cluster it was related to. Right now the cluster name is gleaned from the StrimziUser's annotation. It would be less magical, and therefore easier to debug, if this configuration was more explicit on the StrimziSchemaRegistry itself. For example:
apiVersion: roundtable.lsst.codes/v1beta1
kind: StrimziSchemaRegistry
metadata:
name: confluent-schema-registry
labels:
strimzi.io/cluster: events
spec:
listener: tls
Potentially the KafkaUser should even be configured explicitly too, rather than being the same name as the StrimziSchemaRegistry resource. I'm not aware of an annotation for this; potentially it could be a configuration (in the spec
) instead.
Hi, I'm having trouble deploying to a cluster with scram-sha-512
authentication.
It appears your code tries to construct a client_secret
using the secret generated for a user.
[2022-11-18 18:04:15,550] kopf.objects [INFO ] [kafka-development/confluent-schema-registry] Retrieved cluster CA certificate
[2022-11-18 18:04:15,552] kopf.objects [INFO ] [kafka-development/confluent-schema-registry] Cluster CA certificate version: 11917675
[2022-11-18 18:04:15,566] kopf.objects [INFO ] [kafka-development/confluent-schema-registry] Retrieved cluster CA certificate
[2022-11-18 18:04:15,569] kopf.objects [INFO ] [kafka-development/confluent-schema-registry] Client certification version: 12627686
[2022-11-18 18:04:15,570] kopf.objects [ERROR ] [kafka-development/confluent-schema-registry] Handler 'create_registry' failed with an exception. Will retry.
Traceback (most recent call last):
File "/opt/venv/lib/python3.10/site-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once
result = await invoke_handler(
File "/opt/venv/lib/python3.10/site-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler
result = await invocation.invoke(
File "/opt/venv/lib/python3.10/site-packages/kopf/_core/actions/invocation.py", line 139, in invoke
await asyncio.shield(future) # slightly expensive: creates tasks
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/venv/lib/python3.10/site-packages/strimziregistryoperator/handlers/createregistry.py", line 136, in create_registry
secret = create_secret(
File "/opt/venv/lib/python3.10/site-packages/strimziregistryoperator/certprocessor.py", line 82, in create_secret
client_ca_cert = decode_secret_field(client_secret["data"]["ca.crt"])
KeyError: 'ca.crt'
This secret does not contain a ca.crt
, or user.crt
or user.key
field. It looks like this:
{
"password": "blah13434567",
"sasl.jaas.config": "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"confluent-schema-registry\" password=\"blah13434567\";"
}
And is created using:
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
name: confluent-schema-registry
namespace: kafka-${var.environment}
labels:
strimzi.io/cluster: data-cloud
spec:
authentication:
type: scram-sha-512
I think the code failing is here:
if client_secret is None:
client_secret = get_secret(
namespace=namespace, name=kafka_username, k8s_client=k8s_client
)
logger.info("Retrieved cluster CA certificate")
client_secret_version = client_secret["metadata"]["resourceVersion"]
logger.info(f"Client certification version: {client_secret_version}")
client_ca_cert = decode_secret_field(client_secret["data"]["ca.crt"])
client_cert = decode_secret_field(client_secret["data"]["user.crt"])
client_key = decode_secret_field(client_secret["data"]["user.key"])
Is this approach supported? I'm not using TLS on the listener I'd like the schema registry to use:
apiVersion: roundtable.lsst.codes/v1beta1
kind: StrimziSchemaRegistry
metadata:
name: confluent-schema-registry
namespace: kafka-${var.environment}
spec:
strimziVersion: v1beta2
listener: plain
securityProtocol: SASL_PLAINTEXT
compatibilityLevel: forward
registryImage: confluentinc/cp-schema-registry
registryImageTag: "7.2.1"
state.py
hardcodes the Kafka broker name and namespace to be "events":
strimzi-registry-operator/strimziregistryoperator/state.py
Lines 4 to 14 in 652abb9
I'm working with a namespace of "kafka" and a cluster named "alert-broker". As a result, my strimzi-registry-operator is crash looping with an auth error:
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '5fb4d27a-dc51-4426-ad97-8a0779c67aa4', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '82ffa779-dd6e-4979-9073-6d6527a12a67', 'X-Kubernetes-Pf-Prioritylevel-Uid': '7ab3ebaf-3c64-43ed-accc-590d562d6d4d', 'Date': 'Wed, 20 Oct 2021 16:32:35 GMT', 'Content-Length': '429'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"strimzischemaregistries.roundtable.lsst.codes is forbidden: User \"system:serviceaccount:kafka:strimzi-registry-operator\" cannot list resource \"strimzischemaregistries\" in API group \"roundtable.lsst.codes\" in the namespace \"events\"","reason":"Forbidden","details":{"group":"roundtable.lsst.codes","kind":"strimzischemaregistries"},"code":403}
Highlighting the relevant bit:
strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:kafka:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "events"
As the todo notes, this ought to be configurable so the operator can be used with different namespaces and broker names.
[2023-01-16 04:01:25,818] kopf.objects [ERROR ] [kmon-k1670/confluent-schema-registry] Handler 'create_registry' failed with an exception. Will retry. Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/kopf/_core/actions/execution.py", line 279, in execute_handler_once result = await invoke_handler( File "/opt/venv/lib/python3.10/site-packages/kopf/_core/actions/execution.py", line 374, in invoke_handler result = await invocation.invoke( File "/opt/venv/lib/python3.10/site-packages/kopf/_core/actions/invocation.py", line 139, in invoke await asyncio.shield(future) # slightly expensive: creates tasks File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) File "/opt/venv/lib/python3.10/site-packages/strimziregistryoperator/handlers/createregistry.py", line 131, in create_registry bootstrap_server = get_kafka_bootstrap_server( File "/opt/venv/lib/python3.10/site-packages/strimziregistryoperator/deployments.py", line 83, in get_kafka_bootstrap_server raise kopf.Error(msg, delay=10) AttributeError: module 'kopf' has no attribute 'Error'
I noticed that we are using an old version of kopf
(0.21
).
And this version depends on some deprecated packages.
So, is there any plan to upgrade to the latest kopf
?
Setup:
Minikube 1.27.0
Kubernetes 1.23.
Strimzi 0.13.1 installed via OperatorHub
KafkaCluster "kafka" created in namespace "moonraker"
registry-schemas KafkaTopic in namespace "moonraker" ready
confluent-schema-registry KafkaUser in namespace "moonraker" ready
Install operator via:
helm install -n operators schema-registry lsstsqre/strimzi-registry-operator --set clusterName="kafka",clusterNamespace="moonraker"
Logs from the operator pod:
[2022-10-09 14:49:36,310] kubernetes.client.re [DEBUG ] response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"strimzischemaregistries.roundtable.lsst.codes is forbidden: User \"system:serviceaccount:operators:strimzi-registry-operator\" cannot list resource \"strimzischemaregistries\" in API group \"roundtable.lsst.codes\" in the namespace \"moonraker\"","reason":"Forbidden","details":{"group":"roundtable.lsst.codes","kind":"strimzischemaregistries"},"code":403}
[2022-10-09 14:49:36,313] kopf._core.reactor.r [DEBUG ] Starting Kopf 1.35.6.
[2022-10-09 14:49:36,313] kopf._core.engines.a [INFO ] Initial authentication has been initiated.
[2022-10-09 14:49:36,313] kopf.activities.auth [DEBUG ] Activity 'login_via_client' is invoked.
[2022-10-09 14:49:36,314] kopf.activities.auth [DEBUG ] Client is configured in cluster with service account.
[2022-10-09 14:49:36,315] kopf.activities.auth [INFO ] Activity 'login_via_client' succeeded.
[2022-10-09 14:49:36,315] kopf._core.engines.a [INFO ] Initial authentication has finished.
[2022-10-09 14:49:36,330] kopf._cogs.clients.w [DEBUG ] Starting the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.
[2022-10-09 14:49:36,331] kopf._cogs.clients.w [DEBUG ] Stopping the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide.
[2022-10-09 14:49:36,332] kopf._core.reactor.o [WARNING ] Not enough permissions to list namespaces. Falling back to a list of namespaces which are assumed to exist: {'moonraker'}
[2022-10-09 14:49:36,332] kopf._cogs.clients.w [DEBUG ] Starting the watch-stream for namespaces.v1 cluster-wide.
[2022-10-09 14:49:36,332] kopf._cogs.clients.w [DEBUG ] Starting the watch-stream for strimzischemaregistries.v1beta1.roundtable.lsst.codes in 'moonraker'.
[2022-10-09 14:49:36,333] kopf._cogs.clients.w [DEBUG ] Starting the watch-stream for secrets.v1 in 'moonraker'.
[2022-10-09 14:49:36,333] kopf._cogs.clients.w [DEBUG ] Stopping the watch-stream for namespaces.v1 cluster-wide.
[2022-10-09 14:49:36,334] kopf._cogs.clients.w [DEBUG ] Stopping the watch-stream for strimzischemaregistries.v1beta1.roundtable.lsst.codes in 'moonraker'.
[2022-10-09 14:49:36,334] kopf._cogs.clients.w [DEBUG ] Stopping the watch-stream for secrets.v1 in 'moonraker'.
[2022-10-09 14:49:36,334] kopf._core.reactor.o [WARNING ] Not enough permissions to watch for resources: changes (creation/deletion/updates) will not be noticed; the resources are only refreshed on operator restarts.
[2022-10-09 14:49:36,335] kopf._core.reactor.o [WARNING ] Not enough permissions to watch for namespaces: changes (deletion/creation) will not be noticed; the namespaces are only refreshed on operator restarts.
[2022-10-09 14:49:36,335] kopf._core.reactor.o [ERROR ] Watcher for strimzischemaregistries.v1beta1.roundtable.lsst.codes@moonraker has failed: ('strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "moonraker"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "moonraker"', 'reason': 'Forbidden', 'details': {'group': 'roundtable.lsst.codes', 'kind': 'strimzischemaregistries'}, 'code': 403})
Traceback (most recent call last):
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
response.raise_for_status()
File "/opt/venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.96.0.1:443/apis/roundtable.lsst.codes/v1beta1/namespaces/moonraker/strimzischemaregistries')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
await coro
File "/opt/venv/lib/python3.10/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
async for raw_event in stream:
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/watching.py", line 82, in infinite_watch
async for raw_event in stream:
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/watching.py", line 159, in continuous_watch
objs, resource_version = await fetching.list_objs(
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/fetching.py", line 28, in list_objs
rsp = await api.get(
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 111, in get
response = await request(
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
await errors.check_response(response) # but do not parse it!
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIForbiddenError: ('strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "moonraker"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'strimzischemaregistries.roundtable.lsst.codes is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "strimzischemaregistries" in API group "roundtable.lsst.codes" in the namespace "moonraker"', 'reason': 'Forbidden', 'details': {'group': 'roundtable.lsst.codes', 'kind': 'strimzischemaregistries'}, 'code': 403})
[2022-10-09 14:49:36,336] kopf._core.reactor.o [ERROR ] Watcher for secrets.v1@moonraker has failed: ('secrets is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "secrets" in API group "" in the namespace "moonraker"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'secrets is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "secrets" in API group "" in the namespace "moonraker"', 'reason': 'Forbidden', 'details': {'kind': 'secrets'}, 'code': 403})
Traceback (most recent call last):
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
response.raise_for_status()
File "/opt/venv/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 1004, in raise_for_status
raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 403, message='Forbidden', url=URL('https://10.96.0.1:443/api/v1/namespaces/moonraker/secrets')
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/aiokits/aiotasks.py", line 108, in guard
await coro
File "/opt/venv/lib/python3.10/site-packages/kopf/_core/reactor/queueing.py", line 175, in watcher
async for raw_event in stream:
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/watching.py", line 82, in infinite_watch
async for raw_event in stream:
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/watching.py", line 159, in continuous_watch
objs, resource_version = await fetching.list_objs(
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/fetching.py", line 28, in list_objs
rsp = await api.get(
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 111, in get
response = await request(
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
return await fn(*args, **kwargs, context=context)
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/api.py", line 85, in request
await errors.check_response(response) # but do not parse it!
File "/opt/venv/lib/python3.10/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIForbiddenError: ('secrets is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "secrets" in API group "" in the namespace "moonraker"', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'secrets is forbidden: User "system:serviceaccount:operators:strimzi-registry-operator" cannot list resource "secrets" in API group "" in the namespace "moonraker"', 'reason': 'Forbidden', 'details': {'kind': 'secrets'}, 'code': 403})
It seems the created ServiceAccount doesn't have the permissions it needs to do what it needs to do.
Confluent Schema Registry has a restrictive license, but karapace is an API and Kafka store compatible replacement, licensed more permissively. For some environments this could be desirable.
I am using strimzi-registry-operator to create Schema Registry service in 1 pod and it is running fine. Now I want to distribute Schema Registry to cluster mode to handle fail over.
For Helm chart for operator and registry itself it might be truly helpful to support additional keys for pods, like:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.