mayadata-io / d-operators Goto Github PK

Problem Statement: Of particular interest is a snapshot taker that that keep a rolling window of snapshots on a volume (like once a minute, keep 10 minutes worth, that sort of thing). That would be really helpful for people who have apps that are tightly coupled to the data (e.g. you roll out your new db migrations and hose the database, so you back the code out but the db is still hosed right? Ahh, but you have a 10 minute snapshot!)

For examples PRs like this https://github.com/mayadata-io/oep-e2e-gcp/pull/88/files should be avoided completely.

• A lot of info that goes in here are redundant.
• The coverage report need to make use of custom resource(s) that are generated dynamically after running the tests.

In this issue I propose a design to build desired k8s resources that can be selected by their versions &/or feature flags.

NOTE: Let us take the case of k8s resource with kind Foo
NOTE: In real world, Foo can be replaced by k8s resource e.g. Deployment, ClusterRole, etc

// the folder structure might look like this
projectroot
|_ controller
  |_ my-cool-controller/
    |_ reconciler.go
    |_ reconciler_test.go
|_ common
  |_ selector/
    |_ selector.go
|_ resources
   |_ foo
     |_ foo.go
     |_ foo_test.go
|_ types
  |_ foo.go
  |_ foo_test.go

// in common/selector/selector.go
type TagOperator string

const (
  TagOperatorOR TagOperator = "OR"

  TagOperatorAND TagOperator = "AND"
)

type Tag struct {
  Keys []string
  Operator TagOperator
}

type Selection struct {
  Version string
  FeatureFlag string
  Tag Tag
}

type Terms struct {
  Versions map[string]bool
  FeatureFlags map[string]bool
  Tags map[string]bool
}

func (s Selection) IsMatch(terms Terms) bool {
  // ...
}

// in resources/foo/foo.go

import (
  "project/common/selector"
)

type Config struct {
  Selector selector.Selection
  DesiredName string
  DesiredNamespace string
  DesiredLabels map[string]string
  DesiredAnnotations map[string]string
}

type Foo struct {
  selector selector.Selection
}

func New(config Config) *Foo {
  return &Foo {selector: config.Selector}
}

func (f *Foo) IsSelectionMatch(terms selector.Terms) bool {
 return f.selector.IsMatch(terms)
}

func (f *Foo) GetDesired() (*unstructured.Unstructured, error) {
  fns := []func() (*unstructured.Unstructured, bool) {
    f.getDesiredTypeA,
  }
  for _, fn := range fns {
    desired, ok := fn()
    if ok {
      return desired, nil
    }
  }
  return nil, errors.Errorf("Can't find desired Foo with conf %+v", f.conf)
}

func (f *Foo) getDesiredTypeA() (*unstructured.Unstructured, bool) {
  supportedVersions := map[string]bool{
    "1.9", true,
  }
  supportedFeatureFlags := map[string]bool{
    "alpha": true,
    "beta": true,
    "production": true,
  }
  supportedTags := map[string]bool {
    "linux": true,
    "ubuntu-18.04": true,
  }
  terms := selector.Terms{
    Versions: supportedVersions,
    FeatureFlags: supportedFlags,
    Tags: supportedTags,
  }
  if !f.IsSelectionMatch(terms) {
    return nil, false
  }
  return &unstructured.Unstructured{
    Object: map[string]interface{}{
      "kind": Foo,
      "apiVersion": "example.io/v1alpha1",
    },
  }, nil
}

Thoughts 0

• kind Menu
  • Menu will be a K8s CR with a backing controller
  • Menu can list the Recipes that needs to be run
  • Menu can enable or disable one or more Recipe(s)
  • Menu is equivalent to test suite in testing world
  • NOTE: Menu is not necessarily tied to testing
• kind Recipe
  • Recipe will be a K8s CR with a backing controller
  • Recipe will consist of various actions that needs to be run
  • Recipe is equivalent to test case in testing world
  • NOTE: Recipe is not necessarily tied to testing
• Action
  • Action is not a K8s CR
  • An action can be any operation against a K8s resource
  • An action can be either of following:
    • create, apply, delete, assert, teardown, & more
  • When action is not backed by a K8s resource, we can have a jsonnet action
  • A jsonnet action can be an entire k8s controller in itself (refer MetaController)
  • NOTE: jsonnet action should solve the basic programming constructs only
  • NOTE: We should move from jsonnet action to specific K8s CustomResource eventually
• kind RecipeItem
  • RecipeItem will be a K8s CR without a backing controller
  • RecipeItem can consist of one or more Actions
  • RecipeItem can be referred to by one or more Recipe
  • RecipeItem will not have a backing controller
• kind RecipeValue
  • RecipeValue will be a K8s CR without a backing controller
  • RecipeValue can hold user values that can be used by jsonnet actions
  • RecipeValue can be used to store runtime values while executing jsonnet actions
  • RecipeValue can be used as a config to Recipe
• Assumptions
  • Pure declarative approach
  • No use of templating
  • No use of smart programming syntax
  • Use jsonnet i.e. (json that can embed code) when need for syntax arises
  • Actions of a Recipe can be run in parallel or serial
  • Use additional CR(s) & their backing controllers to make Recipe(s) easy
• Advantage:
  • Declarative intents should help in reducing the effort to build Recipes
  • No bash in yaml
  • No code in yaml
  • One can build a e2e test case in few minutes
  • One can build a install platform in few minutes

References

• Metacontroller

ProblemStatement: As an admin, I would like to expose log errors from any binary running as a Pod as K8s events. I want to avoid viewing of logs.

https://docs.google.com/document/d/1CzQD1MEQDJ3Hdqlw7ihhCinZ2ytIl9a9va5IVrYH05c/

refer - https://github.com/RobertKrawitz/OpenShift4-tools/blob/master/clusterbuster

ProblemStatement: As a DevOps admin, I want to implement my use case as an imperative set of tasks. I would like this to be executed similar to kubernetes Job. In other words, my use case need to run till completion. In addition, I want to avoid coding (i.e. avoid programming) this imperative logic and instead want to use some declarative form to achieve the same.

Ref: https://github.com/argoproj/argo/blob/master/examples/dag-diamond-steps.yaml

• The workflow creation takes into account the parallel/sequential sub-flows or experiments (target could be one application or multiple, each mapped to the respective sub-flow)
• Ability to launch the liveness controllers described here
• Calculate a resiliency score on the basis of outcome of the workflow
• Expose the resiliency data as a result resource

In this issue, we shall design various custom resources that will eventually build DMAAS (data management as a service)

ProblemStatement: As a DevOps engineer, I want a k8s controller that accepts values from various sources and provides a final set of merged values that can in turn be used to arrive at a desired state of a Pod.

For example:

• PodConfig A provides taints & tolerations for Pod
• PodConfig B provides resource limits (memory & cpu) for Pod
• PodConfig C provides environment variables for containers of the Pod
• PodConfigResult F should merge all the above & present a final state that is a 3 way merge of all the above based on priority.

Now there can be custom controllers that make use of resource F to build the desired Pod state & subsequently apply this Pod state against the kubernetes cluster.

Draft specs:

kind: Log
spec:
  podName:
  podNamespace:
  containerSelector:
  tailCount:
status:

• The backing controller fetches the logs from relevant container(s) and sets them against status field

References:

• https://github.com/wercker/stern

ProblemStatement: As an Kubernetes engineer, I would like to extract values from one or more custom resources & use them to build other custom or native resources. I would like to have a kubernetes native solution instead of templating & out of band solutions.

After the Local PV is created and node affinity is set
based on kubernetes.io/hostname label, either:

• hostname label changed on the node or
• underlying storage (device) moved to a new node.

The nodeAffinity parameter on Local PV is not editable,
so to change the Local PV to new node some manual steps are
involved:

• Scale down the deployment or sts using the PVC.
• Save the PVC and PV yamls files.
• Delete the PVC and PV
• Modify the saved PV yaml will the node hostname and apply
  Note: when re-applying the yamls, the uuid of pv and pvc objects will change, so the metadata around self-UUID,etc needs to be cleared.
• Modify the saved PVC yaml for stale references and re-apply.
• Update the PV yaml with the uuid of the newly created PVC
• Scale up the deployment

ProblemSatement: As an automation engineer I would like to implement a litmus testcase to verify creation of cstorpool via Director's recommendations API. I want to avoid kubectl, grep, awk, etc in my testcase. I want the testcase should be declarative without use of templating, etc. This testcase should be easy to write & understand. I want the intent to be as close as possible to the business/test logic without exposing technical aspects e.g. http status code, retries, body format, etc.

Refer: mayadata-io/oep-e2e#439
TestCaseID: TCID-DIR-OP-CSP-REC-CREATE-MIRROR
Following are the steps performed:

- use of secret to fetch Director user & pw
- config to provide Director info e.g. ip address, group id, etc
- verify openebs components to be in running state
- director api to verify its components in running state
- director api to fetch recommendations
- fetch recommendation id
- director api to list capacity recommendations
- director api to list device recommendations
- director api to create cstor pool
- director api to fetch cstorpool operation id
- director api to execute cstor pool
- director api to wait for successful creation of cstor pool

ProblemStatement: A statefulset can be deployed with multiple replicas. In addition, each replica can be configured to use one or more persistent volumes. While due care can be taken to spread the statefulset application replicas across zones or nodes, similar strategy needs to be followed for these replicas' backing volume(s). It will be good to ensure these volumes are spread out & are never placed on a single node or single zone.

Possible Solution: Specific statefulsets can be watched (via k8s controllers) & their corresponding volume claims can be tracked. The volume claims can further be filtered based on the storage class. Once required details are achieved, these information can be saved in a new custom resource. This should ensure storage provisioners to run through their volume placement decisions.

# This custom resource will select statefulset whose volume info need
# to tracked & saved
kind: StatefulsetVolumePolicy
spec:
   # select a StatefulSet based on following criterias
   select:
     matchLabels:
     matchAnnotations:
     matchFields:
   placement:
     # spread will spread the volumes across zones or across hosts or
     # will get placed based on corresponding application pods
     spread: # Zone, Host, Pod
     # topology can be used to further determine which zones, hosts, etc
     # these volumes should get placed
     topology:
       hosts:
       - host 2
       - host 3

kind: VolumeGroup
spec:
   refer:
      kind: StatefulSet
      apiVersion: apps/v1
      name: my-sts
      namespace: my-ns
  volumes:
  - pvcName:
    podName:
    podNodeName:

ProblemStatement: As a DevOps admin, I need to create, delete & re-create the resource with some modifications. This workflow is executed in a schedule.

Background

• One of the requirements in any chaos engineering framework is to provide an effective means of carrying out health checks (synonymous with jargons like steady-state checks, pre/post chaos checks, entry/exit criteria, "liveness" checks etc.,) specific to the object under test. The common requirement of such checks is to Effectively relay information about the availability, and optimal performance of the application or infra under test. An ideal liveness checks would to do so even as the chaos/fault is in progress (rather than give a one-rime indicator before and after fault - as this may not account for transient errors and recovery).

• In litmus, most checks fall under the former category (one time command-based investigation before and after chaos) and few in the latter (continuous check during chaos). These are executed as jobs executing a bash/python script running in perpetual mode (infinite loops) that are launched and killed by the experiment logic. Typically these take some service endpoints, polling intervals, retry counts/timeouts as inputs to perform the checks.

• Some references:

  • MySQL Liveness Checker
  • NFS Liveness Checker

Problem Statement

As is apparent, these liveness checks are very simplistic and probably do not provide an entirely accurate picture of availability, at least from the experiment standpoint. As cases in point:

• The MySQL checker performs a status query to check "health". However, from a user context, the Key Performance Indicators (KPIs) associated with the MySQL deployment may be far more than just this. It could be the min transactions/queries per seconds, the count in the slow query log, error counts etc.., When Chaos is performed, on say, a MySQL statefulset, the liveness checker should look for these attributes and relay the "liveness" summary as, say, a metric which the experiment can consume to determine resiliency. Of course, such a checker will expose multiple "optional" indicators which the SRE/developer may choose to consume or ignore (a status check might be sufficient)

• The NFS checker does a showmount command that just verifies if the NFS provisioner continues to export a certain path (which contains a openebs volume string). It does not mention if the NFS volumes (which internally uses OpenEBS PVs) are writable, if the provisioner continues to support creation and deletion of new PVs, if the NFS volumes exposed continue to offer a desired level of QoS, if data integrity is maintained across chaos (the experiment logic does this today in a "manual" fashion by reading data/performing md5sums after chaos), etc.,

Requirement

• Metac, which offers great scope in terms of low turnaround/development times and flexible custom resource definitions can be used to create improved "liveness controllers", on per application basis, with each controller associated with rich, but focused schema for the respective application. These controllers themselves can be shipped with the litmus chaos operator, with the lifecycle of the associated custom resources being managed by the experiment logic, much like how it does today for the previously described liveness job/pods specs. These "liveness controllers" can also be used in a standalone capacity (not part of a litmus chaos experiment, per se) in a production system or used by other chaos frameworks.

• While the above description sounds like it might have a bit of an overlap with traditional existing monitoring solutions which use exporters and alert rules -- there is no one integrated way today to use app information to decide on it's "health". At least not w/o much plumbing on the part of the SREs. These controllers can fill an important gap.

Requirement:

• The users workflow to trigger/abort chaos should be as simple as annotating and de-annotating his application deployment.
• In other words, ChaosEngine resources should be created (triggering the experiment) and removed (aborting the experiment) upon adding an experiment specific annotation. For example: litmuschaos.io/pod-delete: true | false (or removal).
• This can be extended to handle repeated/scheduled execution, say litmuschaos.io/pod-delete-schedule: <time units> which might create a schedule custom resource (the scheduler controller then creating the engine)

Considerations:

• The chaosengine spec is today holds (a) app info / (b) experiment info / (c) run-properties. Out of these (c) and to an extent (b) are static information as far as a user/SRE is concerned and can be derived from templates the user can pull. The app info (a) is desired to be pulled by the controller under discussion: as part of generating the chaosengine attachment.

• Schedule are subject to halt/resumption. Lets say that is handled by other controllers (scheduler, as discussed above). In such cases the current engine/schedule generator should be aware and reconcile accordingly.

refer - https://github.com/kubernetes/kube-openapi/pull/110/files

ProblemStatement: As an automation engineer I want to pre-pull images to my cluster as an one time activity. I would like to pre-pull images to all or selected nodes. In addition, there are several other scenarios where I would like to run pods to completion on several nodes.

Refer

• https://github.com/AmitKumarDas/metac/tree/master/examples/daemonjob
• http://blog.itaysk.com/2017/12/26/the-single-use-daemonset-pattern-and-prepulling-images-in-kubernetes

ProblemStatement: As an automation engineer, I want to upload specific kubernetes resources as well as pod logs from a running kubernetes cluster to some github repo. This helps me to refer to these resources long after the kubernetes cluster is gone.

Some references:

• https://github.com/piontec/netperf-operator
• https://github.com/portworx/fio-tools
• https://github.com/pingcap/chaos-mesh/tree/master/controllers/iochaos
• https://github.com/rancher/system-upgrade-controller

• In our test cases we are installing openEBS and after installing openEBS we need to uninstall it.
• Uninstallation of openebs includes multiple tasks and after that also few times it gets stuck in terminating state.

• While implementing test cases in oep-e2e we are performing same kind of steps multiple times these all redundant tasks need to be removed.
• There are many tasks which we are performing in almost all test cases.
  For Ex:-
  • Openebs components are in running state or not.
  • Check whether the resource is present or not.
  • Check the state of the job.

ProblemStatement: As a k8s engineer I want to list various kubernetes resources (custom or native) based on labels, &/or annotations. I would then like to apply new custom labels &/or annotations against these selected resources.

ProblemStatement: As a DevOps admin, I want to list down the pain points associated with litmus scripts. I would like to solve them via tools or enhancing litmus itself. This issue will acts as a tracker for the same.

litmuschaos/litmus#1223

ProblemStatement: ResourcePipe is a custom resource that extracts values from one or more k8s resources and manages to build a new k8s resource from these extracted values & apply this new resource in the k8s cluster.

kind: ResourcePipe
spec:
  input:
  - apiVersion:
    resource:
    name:
    namespace:
    get:
    - key: # should unique across all gets
      # json path that uses https://github.com/tidwall/gjson
      path: 
      # string, int64, []string, []int64, []map[string]interface{}, map[string]string
      datatype: # defaults to string
      value: # optional
        default: # default value if path does not have any value
        prefix:   # if set final value will have this prefix
        suffix:   # if set final value will have this suffix
  output:
    # default resource
    template:
    # no of replicas of this template that get applied against
    # k8s cluster; is optional & defaults to 1; 0 is invalid
    replicas:
    # above template is updated with these set options
    set:
    # use the value extracted from source path & set it against
    # the destination path mapped by the key
    - key: # should match against one present in input[*].get[*].key
      path:
    # multiple keys can be used against a single path if these keys' values
    # belong to same data type
    # When multiple keys are used their corresponding values are merged
    # and set against the destination path
    - key: 
      path:

ProblemStatement: A report card that provides various results w.r.t Recipe custom resources will be helpful. This report card can have passed, failed count. It can optionally have the detailed fail messages. This report card should be able to opt in or out Recipes based on label / annotation / name selectors.

Refer - https://github.com/zebardy/openebs-jiva-RW-check

As a developer I would like to make http API calls declarative. The http API invocation & its result can form the basis of a new Kubernetes custom resource.

Sample http based request & response

curl -u "${CATTLE_ACCESS_KEY}:${CATTLE_SECRET_KEY}" \
-X POST \
-H 'Accept: application/json' \
-H 'Content-Type: application/json' \
-d '{"clusterId":"1c2", "raidGroupConfig":{"type":"mirror", "groupDeviceCount":2}}' \
'http://34.69.231.86/v3/groups/1a11/recommendations/1r!cstorpool/?action=getcapacityrecommendation'

HTTP/1.1 POST /v3/groups/1a11/recommendations/1r!cstorpool/?action=getcapacityrecommendation
Host: 34.69.231.86
Accept: application/json
Content-Type: application/json
Content-Length: 76

{
"clusterId": "1c2",
"raidGroupConfig": {
"type": "mirror",
"groupDeviceCount": 2
}
}

HTTP/1.1 200
date: Thu, 05 Mar 2020 05:10:55 GMT
x-api-user-id: 1a9
x-api-client-ip: 49.37.204.116
x-api-roles: ProjectOwner
vary: Accept-Encoding
content-type: application/json; charset=utf-8
transfer-encoding: chunked
x-api-account-id: 1a11
x-api-schemas: http://34.69.231.86/v3/groups/1a11/schemas
connection: keep-alive
content-encoding: gzip
expires: Thu, 01 Jan 1970 00:00:00 GMT

{
"type": "collection",
"resourceType": "recommendation",
"links": {
"self": "…/v3/groups/1a11/recommendations/1r!cstorpool/"
},
"createTypes": { },
"actions": { },
"data": [ 2 items
{
"id": null,
"type": "cStorPoolCapacityRecommendationOutput",
"links": { },
"actions": { },
"baseType": "cStorPoolCapacityRecommendationOutput",
"deviceGroupName": "HDD",
"maxCapacity": "536870912k",
"minCapacity": "53687091200"
},
{
"id": null,
"type": "cStorPoolCapacityRecommendationOutput",
"links": { },
"actions": { },
"baseType": "cStorPoolCapacityRecommendationOutput",
"deviceGroupName": "HDD-512",
"maxCapacity": "536870912k",
"minCapacity": "53687091200"
}
],
"sortLinks": { },
"pagination": null,
"sort": null,
"filters": { },
"createDefaults": { }
}

Figure out a way/approach to extract value from specific fields in a declarative manner.

One of the approaches that we might want to follow is to find the diff of the declared indent against the actual specifications. We might probably be able to get the specific field's actual value by comparing it against a dummy value.

For example for following declaration, we should be able to find the actual status.phase value of my-pod:

kind: Pod
apiVersion: v1
metadata:
  name: my-pod
status:
  phase: extract-n-register

References:

• https://github.com/go-test/deep
• https://github.com/kr/pretty/blob/main/diff_test.go
• https://github.com/kylelemons/godebug

Alternative

Use jsonnet to do all of the above. Weigh in the pros & cons & proceed. Jsonnet as a controller needs to be supported any way.

Problem Statement: I would like to have a simple declarative way to manage execution of various Kubernetes resources (native as well as custom).

kind: Run
spec:
  onError:
    retry:
      times: # some int value
      interval: # some timeperiod e.g. 2 secs, 1 min, etc
    abort: # if true will not retry
  tasks:
  - desc: # description of this task
    id: # some id given to this task, must be unique in this Run resource

    # state can be used to create / apply it in the cluster
    # state along with delete field can be used to delete the state from cluster
    # state can embed an entire K8s resource _(native or custom)_
    state: # map[string]interface{}

    # select one or more resources & execute assert or delete action
    # can use either `state` or `select` to execute assert or delete action
    # both state & select can not be used together
    select:

    # assert acts as an action similar to create, delete, etc
    # assert acts as a condition when used along with create, delete, etc
    # assert can be used with select to filter resource & run assertion
    assert:
    - refTask: # taskid; assert is based on this task; optional
      key: # json paths, etc. optional
      operator: # Equals, NotEquals, EqualsCount
      value: # string or int value

    replicas: # when more than one states need to be maintained/created

    delete: # optional; delete can also get triggered if state.spec is set to nil
      operator: # RandomCount, etc.
      value: # any string or int value

https://github.com/stevelacy/kuberhaus

Problem Statement: Most of my requirements deal with extracting & updating json elements. I don't want to use a full blown programming language to deal with these. Jsonnet solves my purpose very well, is easy to learn, build & test. Over time my effort to build Kubernetes controllers decreases.

Assumption: I would like to build something on top of Metac that already abstracts away Kubernetes controllers.

Goal: This solution should provide further levels of abstraction to build workflows that are native to Kubernetes.

https://velero.io/docs/master/restic/

ProblemStatement: As a kubernetes engineer I would like to install, run & verify one or more applications (that includes required controllers) on my preferred kubernetes cluster. I want a cli that can help me achieve my goal.

NOTE: I do not want to deal with HELM or other tools to achieve this.
NOTE: I do not want to configure yamls to achieve my purpose.
NOTE: This cli should provide me an opinionated solution.

Use Cases

• Ability to test my K8s controller using a mocked kubernetes control plane
• Ability to install, run & test Cassandra on OpenEBS local PV on GKE
• Ability to install & verify CStorPool operator in my E2E pipelines

Notes:

• Mocked k8s control plane implies k8s api server binary, etcd binary & kubectl talking to each other

d-operators make use of Job as a custom resource. It needs to be renamed to Recipe. In addition, the apiVersion should be changed to dope.metacontroller.io/v1

This will also involve changes to package names, variable names, logs, comments, docs and anything that is related to Job.

ProblemSatement: As a DevOps admin I have several applications that depend on environment variables. I would like to expose the available environment variables via a kubernetes custom resource. This would help me implement my tasks in a kubernetes friendly way.

kind: EnvGet
spec:
  include:
  - <regex>
  exclude:
  - <regex>
status:
  phase:
  # will hold environment variable key: value pairs
  variables:

Following is the current schema:

// If defines the conditions required to verify the
// presence of, absence of, equals to & other checks
// against one or more resources observed in the cluster
type If struct {
	// OR-ing or AND-ing of conditions
	IfOperator IfOperator `json:"operator,omitempty"`

	// List of conditions that get executed as part of
	// if operation
	IfConditions []IfCondition `json:"conditions,omitempty"`
}

// IfCondition to match, filter, verify a kubernetes resource.
type IfCondition struct {
	// Selector to filter one or more resources that are expected
	// to be present in the cluster
	ResourceSelector metac.ResourceSelector `json:"resourceSelector,omitempty"`

	// ResourceOperator refers to the operation that gets executed to
	// the selected resources
	//
	// Defaults to 'Exists'
	ResourceOperator ResourceOperator `json:"operator,omitempty"`

	// Count comes into effect when operator is related to count
	// e.g. EqualsCount, GreaterThanEqualTo, LessThanEqualTo.
	Count *int `json:"count,omitempty"`
}

Following is the proposed schema

// ResourceCheck defines the conditions required to verify the
// presence of, absence of, equals to & other checks
// against one or more resources observed in the cluster
type ResourceCheck struct {
	// OR-ing or AND-ing of conditions
	SelectConditionOperator ResourceSelectConditionOperator `json:"selectConditionOperator,omitempty"`

	// List of conditions that get executed as part of
	// if operation
	SelectConditions []ResourceSelectCondition `json:"selectConditions,omitempty"`
}

// ResourceSelectCondition to match, filter, verify a kubernetes resource.
type ResourceSelectCondition struct {
	// Selector to filter one or more resources that are expected
	// to be present in the cluster
	Selector metac.ResourceSelector `json:"resourceSelector,omitempty"`

	// Operator refers to the operation that gets executed to
	// the selected resources
	//
	// Defaults to 'Exists'
	Operator ResourceSelectorOperator `json:"selectOperator,omitempty"`

	// Count comes into effect when operator is related to count
	// e.g. EqualsCount, GreaterThanEqualTo, LessThanEqualTo.
	Count *int `json:"count,omitempty"`
}

ref - https://gist.github.com/jjo/1ccade2ea3ae05c5825fd37c8722d4f6
ref - https://github.com/go-cmd/cmd

Draft - 0

kind: Command
apiVersion: dao.mayadata.io/v1alpha1
metadata:
  name: install-cephfs
  annotations:
    cmd: &cmd apt-get update -qy && apt-get install -qy tgt ceph-fs-common ceph-common xfsprogs
spec:
  hostPID: true
  securityContext:
    privileged: true
  command:
    - nsenter
    - --mount=/proc/1/ns/mnt
    - --
    - sh
    - -c
    - *cmd
status:
  exitCode:
  reason:

Problem Statement: As a developer, I want to reserve NDM exposed BlockDevice resource for use by the operators that are initiated the creation of this BlockDevice.

Possible Solution: In order to achieve this, NDM should reserve the BlockDevice even before this BlockDevice gets discovered. NDM might introduce a new CR called BlockDeviceReserve that will have necessary details of the BlockDevice that is supposed to be discovered in the future. NDM on its part when discovers a BlockDevice can run through additional logic that tries to map the BlockDeviceReserve against this newly discovered BlockDevice. If a match is found, it should mark this BlockDevice as reserved by associating this BlockDevice with BlockDeviceReserve's UID. Operators can then claim the BlockDevice via BlockDeviceClaim with the latter having the BlockDeviceReserve UID.

ProblemStatement: As a tester, I make use of Run resource to provision a volume in my kubernetes cluster. I would like to create 100s of volumes using the same Run resource & subsequently derive various metrics such as average, fastest & slowest creation times. Similarly, average, fastest & slowest deletion times.

refer - https://github.com/bpineau/katafygio

As a developer I want to use better assertion logic than the following:

- name: Fetch OpenEBS control plane pods ready status
  shell: kubectl get pods -n openebs | grep {{ item }} | awk '{print $2}' | awk -F'/' '{print $1}' && kubectl get pods -n openebs | grep {{ item }} | awk '{print $2}' | awk -F'/' '{print $2}'
  register: ready_status
  with_items:
    - '{{ openebs_components }}'
  until: '{{ ready_status.stdout_lines | unique | length == 1 }}'
  retries: 20
  delay: 5

It will be great to assert list of values in a declarative manner.

For example below yaml should be enough to assert any Pods with the given label to be in Running state. It should return error if there are any Pods that are not Running.

kind: Pod
metadata:
  labels:
    name: app
status:
  phase: Running