#53 is a draft PR but the checks still run.

I think we should disable the checks to save resources.

Compute node panicked while starting:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: internal error: parse error unknown field `chunk_size`, there are no fields for key `batch` at line 10 column 1
disabled backtrace', src/common/src/config.rs:51:53
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

This is caused by the latest update of RisingWave: risingwavelabs/risingwave#2686

Need to fix the current config in ./template/compute-config.yaml

As mentioned in the title, we need documents about running tests, especially load tests on the RisingWaves deployed by the operator. Currently, there are no public documents or scripts for running load tests against RisingWave. So I guess we need to wait before there's one.

We can add a kubectl-plugin for risingwave-operator just like kubectl-rw. Users can use it by kubectl rw init to deploy the risingwave-operator && use kubectl rw create instance --params.. to create a risingwave db.

Ref: https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/

When create or delete risingwave, error from server:
Internal error occurred: failed calling webhook "mrisingwave.kb.io": Post "https://127.0.0.1:57355/mutate-risingwave-singularity-data-com-v1alpha1-risingwave?timeout=10s": dial tcp 127.0.0.1:57355: connect: connection refused

I have met in aws eks and kind cluster, maybe webhook network bug?

We need to support to update risingwave which includes:

1. Change image to upgrade
2. Change replicas to scale

Not Supported in the short term:

1. Change resource request and limit(VPA)
2. Change running params

Add integration tests for the RisingWaveController, e.g.,

• Leverage the ActionHook to validate the inputs and outputs of each action
• Tests for controllers under different states, simulated by the fake client, possible test cases:
  • Initializing, with a new RisingWave object
  • Initializing, with partial sub-resources
  • Running, and stable
  • Running, and spec changed
  • Running, and some sub-resources deleted, related to #64
  • Upgrading, and outdated sub-resources
  • Upgrading, and latest sub-resources
  • Deleted object
  • No object
  • ...

NetworkPolicy lets us define the ingress and egress policies for the Pods. We can integrate the NetworkPolicy in the RisingWave CRD to provide controls on the network policies. A typical use case would be configuring the security group of the RisingWave instances.

References:

• NetworkPolicy

need support ETCD for meta node

There is a parameter in risingwave.toml config map:

...
[storage]
data_directory = "hummock_001"

This should be customized to let many RisingWave instances share one bucket

helm/risingwave-operator
helm/risingwave-example

It's essential to provide monitoring tools for logs because it's hard for normal users to interact with Kubernetes directly. Here're some open-source projects that might help:

• fluend
• jaeger
• loki

Run make docker-build will get the following errors if ./vendor directory does not exist.

Step 10/19 : COPY vendor* vendor/
COPY failed: no source files were specified

For now, need to spend too much time when deploying a suite of RisingWave in a new cluster. Maybe can optimize component startup speed by reducing RisingWave image size.

We shouldn't care about how to deploy and operate the MinIO service. I think we can leverage the operator from the community.

https://github.com/minio/operator

If we don't provide spec for compactNode when deploying RisingWave, the operator will panic at: pkg/controllers/risingwave/risingwave_controller.go:262 since *rw.Spec.CompactorNode.Replicas is a nil pointer.

cannot delete compactor node when delete RisingWave

Add tests for the ctrlkit package, e.g.,

• Tests for the OptimizeWorkflow method and make sure every rule works
• Tests for the Group implementations to ensure their semantics
• Tests for the joinResultAndErr method to make sure the result and err are as expected

for now, the compute node --host use the Pod IP, need to change it to PodName.svc-name.namespace.svc.cluster.local.

The RisingWave provides a field specifying the secret containing the etcd credentials. But it doesn't work now. AFAIK, the RisingWave itself hasn't supported authenticating with etcd currently. This is the tracking issue for the operator side, and I'm going to create an issue in the risingwave repo as well.

Add tests for the implementation of RisingWaveControllerManagerImpl, a.k.a. the risingWaveControllerManagerImpl, e.g.,

• Tests for the syncObject method with different inputs, validating the lazy sync mechanism
• Tests for the isObjectSynced method with different inputs, validating the lazy sync mechanism
• Tests for each action implementation to ensure that they don't panic and works

Tips:

• The lazy sync mechanism is mainly based on the risingwave/generation label. If the label value is smaller than the generation of the RisingWave object, then the object should be synced/updated. Otherwise, we do nothing. A special case is that when the label value is nosync, the object won't be synced. Creation always happens when the object isn't found.

Currently, we have the RisingWave CR for managing the RisingWave instances in the Kubernetes. We can manipulate the RisingWave resources with kubectl. I'm thinking of abstracting other resources as CRs, such as the stream sources.

The stream sources should have the connection endpoint, the credentials, and other configurations supported by either the stream provider or the RisingWave. Then we can have a controller that syncs the stream sources with the RisingWave instance and reports the status.

Add tests for the RisingWaveObjectFactory, e.g.,

• Tests for building new ConfigMap for RisingWave config
• Tests for building new Service/Deployment for meta/frontend/compactor
• Tests for building new Service/StatefulSet for compute

Possible checks:

• ownership checks
• labels checks
• spec checks
  • especially the command and args checks for compatibility
• panic checks

Tips:

• Simulating the webhook manually

Currently, I find that we have too few tests to ensure the quality of the operator. We could improve our tests in several aspects:

• Unit test
  • #75
  • #76
  • #77
  • #78
  • #82
• Integration tests
  • #79
• E2E test
  • #80
  • #81

Now it's legal to have a YAML like this, specifying the meta storage type as etcd but not giving the endpoint:

spec:
  metaNode:
    storage: 
      type: ETCD
      etcdEndpoint: 

It's a bad YAML and should be rejected by webhook.

There is an issue running the example psql queries on the risingwave readme on the risingwave cluster. Running the example queries result in an rpc error. This error is not replicated when running the same queries with the docker image docker run -it --pull=always -p 4566:4566 -p 5691:5691 ghcr.io/singularity-data/risingwave:latest playground

Expected Behavior

No errors.

Current Behavior

RPC error which causes the compute node to crash.

Steps to Reproduce

1. Set up the operator as per the risingwave operator readme but use this yaml for the risingwave object. This is similar to the default settings but implements a nodeport to access the frontend node.
2. Enter psql shell

PHOST=`kubectl get node -o=jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}'`
PPORT=`kubectl get service -n test test-risingwave-amd64-frontend-nodeport -o=jsonpath='{.spec.ports[0].nodePort}'`
psql -h PHOST -p PPORT -d dev -U root

3. Run psql queries

/* create a table */
create table t1(v1 int not null);

/* create a materialized view based on the previous table */
create materialized view mv1 as select sum(v1) as sum_v1 from t1;

/* insert some data into the source table */
insert into t1 values (1), (2), (3);

/* (optional) ensure the materialized view has been updated */
flush;

/* the materialized view should reflect the changes in source table */
select * from mv1;):

As title

Currently, the testing script only checks the four services for the liveness of the RisingWave cluster. I think we can improve it by checking the connectivity with a PostgreSQL client.

Related scripts are under the testing dir.

for meta-server, compute node, compactor node, and frontend. TCPSocketAction probe should be added
for minIO, use HTTPGetAction probe

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Add tests for the webhooks, e.g.,

• Tests for the mutation webhook, mainly for the defaulter
• Tests for the validation webhook, e.g.,
  • Tests of unexpected fields, e.g., nil ptr fields, invalid etcd endpoints, unexpected enum values
  • Tests of unexpected updates, such as updating immutable fields

As the project grows, we need to expose metrics about the webhooks and controllers to build the observability & maintenance stacks.

Metrics we'd like to have:

• Webhooks:
  • webhook_request_count, labels:
    • type, the value should be mutating or validating
    • group, the target resource group of the webhook, e.g., risingwave.risingwavelabs.com
    • version, the target API version, e.g., v1alpha1
    • kind, the target API kind, e.g., risingwave, risingwavepodtemplate
    • namespace, the namespace of the object, e.g., default
    • name, the name of the object
    • verb, the verb (action) on the object which triggers the webhook, the value should be one of "create", "update", and "delete".
  • webhook_request_pass_count, with the same labels as webhook_request_count
  • webhook_request_reject_count, with the same labels as webhook_request_count
  • webhook_request_panic_count, with the same labels as webhook_request_count
• Controllers:
  • controller_reconcile_count, labels:
    • group, version, kind mentioned above
    • namespace, name
  • controller_reconcile_requeue_count, with the same labels as controller_reconcile_count
  • controller_reconcile_error_count, with the same labels as controller_reconcile_count
  • controller_reconcile_requeue_after (could be a histogram), with additional labels:
    • after, the duration before the next requeue in milliseconds
  • controller_reconcile_duration, the time elapsed during the reconciliation, with the same labels as controller_reconcile_count
  • controller_reconcile_panic_count, with the same labels as controller_reconcile_count

The collectors of these metrics can all be implemented by using a proxy pattern.

The Kubernetes community has been working on the auto scaler for a long time. They have had the HorizontalPodAutoScaler GA and provided a CRD and the controller of the VerticalPodAutoScaler. These auto-scalers targets the workload resources such as Deployment and StatefulSet. But we will scale the RisingWave instance, which contains four components and multiple groups of workload resources. Therefore, there're two options for implementing the auto-scaling:

• Leverage the pod auto-scaler provided by Kubernetes and ignores the replicas defined in the RisingWave spec when the auto-scalers are enabled. We need to define the behavior when auto-scaler is enabled/disabled and ensure there're no ambiguities.
• Define new CRs ourselves and re-implement the trigger and scaling progress by operating the RisingWave resources directly. This lets us have more control over the progress, e.g., we can have a customized policy for triggering the scaling, and there will be no ambiguity, but we need to do a lot of work that the community has done.

Either option makes sense to me. Since we don't have much effort, I think we have to decide what to do.

References:

• HorizontalPodAutoScaler
• VerticalPodAutoScaler

as title

After merging #61, the E2E test is modified to test against an in-memory RisingWave cluster, which is not comprehensive enough. We should test against all the storages types, including:

• memory/etcd based meta
• memory/MinIO/S3 based compute and compactor

We should ensure that it works in any combination.

We can track the kubectl-plugin commands development progress here.

• Install
• Uninstall
• Create
• Delete
• List
• Scale
• Config
• Describe
• Stop
• Resume
• Restart
• Update
• Upgrade
• Doc for kubectl-plugin
• Publish kubectl-plugin to the krew https://krew.sigs.k8s.io/docs/developer-guide/release/new-plugin/

We recommend to support monitoring based on prometheus-operator (https://github.com/prometheus-operator/prometheus-operator)

Need to support install compactor (as statefulset)
Ref: https://github.com/singularity-data/risingwave/blob/main/docker/docker-compose.yml

Operators have many crucial definitions like specs and API interfaces that should be consistent for specific versions, so that other components should confidently rely on.

After several refactors #61 and #109, the functionalities for monitoring are broken, such as:

• Auto provisioning of the ServiceMonitors provided by the Prometheus Operator
• Guidance to start monitoring the RisingWaves in Kubernetes
• Documents about testing against the RisingWaves in Kubernetes

Currently, the operator won't react to problems like nodes hanging or accident deletion of sub-resources such as Deployments, Services, and StatefulSets. We want to make sure that we can recover from these issues.

• Leverage the liveness probe so that we can restart the unhealthy node and recover from it.
• Implement the reactions to the absence of sub-resources, especially when the cluster's running.

Currently, the RisingWave Operator runs a workflow for each reconciliation. The workflow is organized in pure Go with help functions provided by the ctrlkit. It's easy to understand and develop, but it's hard to monitor and trace for now. We need mechanisms for this. Here're the topics I think would be helpful if achieved:

• Monitoring
  • Expose metrics of the workflow status, such as ID, target, result, error, and elapsed time
  • Expose metrics of the workflow action status, such as result, error, and elapsed time
  • Dashboard (panels) for monitoring the running status of the operator workflows, e.g. event rates, avg/p50/p99 time, common errors
• Tracing
  • Design and implement the structures and functions for helping trace the workflow, capturing metrics and states (optional)
  • Provide an option for the tracing and provide a way to get the trace result
  • Dashboard for trace analysis, e.g., the time costs in a tree, a graph where each node(action) is colored by its result
  • Provide an option for capturing the states of the future workflows and a way to replay the workflows with states captured. This is useful when debugging the online service.

References:

• RFC ctrlkit
• The main workflow

Unhandled exception in gopkg.in/yaml.v3

Add tests for the RisingWaveObjectManager, e.g.,

• #99
• Tests for checking if the immutable copy is left unchanged
• Tests for checking if the status is changed in memory

sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:245: expected type *v1.StatefulSet, but watch event object had type *unstructured.Unstructured

This bug occurred after PR #21 merged.

maybe controller-runtime issue.