Giter Club home page Giter Club logo

managed-cluster-config's Introduction

managed-cluster-config repository

This repo contains static configuration specific to a "managed" OpenShift Dedicated (OSD) cluster.

How to use this repo

https://issues.redhat.com/browse/SDE-2786 has change the repo slightly: /deploy holds the sources of truth, and /generated_deploy holds the configurations that will be applied by Hive. To add a new SelectorSyncSet, add your yaml manifest to the deploy dir, then run the make command.

Alternatively you can enable GitHub Actions on your fork and make will be ran automatically. Additionally, the action will create a new commit with the generated files.

To add an ACM (Governance) Policy

make will look for config.yaml files, runs it with the PolicyGenerator binary and save the output to generated_deploy/acm-policies directory. make will then automatically add the policy as a new SelectorySyncSet.

Building

Dependencies

  • oyaml: pip install oyaml

Configuration

All resources in generated_deploy/ are bundled into a template that is used by config management to apply to target "hive" clusters. The configuration is deployed to the "hive" cluster inside a SelectorSyncSet.

SelectorSyncSet deployment supports resources that are synced down to OSD clusters. Each are explained in detail here. The general configuration is managed in a config.yaml file in each deploy directory. Key things of note:

  • This file is now mandatory in the scope of OSD-15267 and have been added to all folders. In case it is not define, make will fail
+ scripts/generate_template.py -t scripts/templates/ -y deploy -d /Users/bdematte/git/managed-cluster-config/hack/ -r managed-cluster-config
ERROR : Missing config.yaml for resource defined in deploy/acm-policies
Some config.yaml files are missing, exiting...
make: *** [generate-hive-templates] Error 1
  • Configuration is not inherited by sub-directories! Every (EVERY) directory in the deploy/ hierarchy must define a config.yaml file.

You must specify a deploymentMode property in config.yaml.

  • deploymentMode (optional, default = "SelectorSyncSet") - either "Policy" or "SelectorSyncSet".

Direct Deployment

Direct deployments to Hive clusters should be done via app-interface.

SelectorSyncSet Deployment

In the config.yaml file you define a top level property selectorSyncSet. Within this configuration is supported for matchLabels, matchExpressions, matchLabelsApplyMode, resourceApplyMode and applyBehavior.

  • matchLabels (optional, default: {}) - adds additional matchLabels conditions to the SelectorSyncSet's clusterDeploymentSelector
  • matchExpressions (optional, default: []) - adds matchExpressions conditions to the SelectoSyncSet's clusterDeploymentSelector
  • resourceApplyMode (optional, default: "Sync") - sets the SelectorSyncSet's resourceApplyMode
  • matchLabelsApplyMode (optional, default: "AND") - When set as "OR" generates a separate SSS per matchLabels conditions. Default behavior creates a single SSS with all matchLabels conditions. This is to tackle a situation where we want to apply configuration for one of many label conditions.
  • applyBehavior (optional, default: None, see hive default) - sets the SelectorSyncSet's applyBehavior

You can also define a top level property policy to specify the behaviour of ./scripts/generate-policy-config.py for the resource. Supported sub-properties :

  • complianceType (optional, default: "mustonlyhave", see operator values - select the compliance type for the policy when used by ./scripts/generate-policy-config.py)
  • metadataComplianceType (optional, default: "musthave", see operator values - select the compliance type for metadata for the policy when used by ./scripts/generate-policy-config.py)

Example to apply a directory for any of a set of label conditions using Upsert:

deploymentMode: "SelectorSyncSet"
selectorSyncSet:
    matchLabels:
        myAwesomeLabel: "some value"
        someOtherLabel: "something else"
    resourceApplyMode: "Upsert"
    matchLabelsApplyMode: "OR"
policy:
    complianceType: "mustonlyhave"
    metadataComplianceType: "musthave"

Selector Sync Sets included in this repo

Prometheus

A set of rules and alerts that SRE requires to ensure a cluster is functioning. There are two categories of rules and alerts found here:

  1. SRE specific, will never be part of OCP
  2. Temporary addition until made part of OCP

Prometheus and Alertmanager persistent storage

Persistent storage is configured using the configmap cluster-monitoring-config, which is read by the cluster-monitoring-operator to generate PersistentVolumeClaims and attach them to the Prometheus and Alertmanager pods.

Curated Operators

Initially OSD will support a subset of operators only. These are managed by patching the OCP shipped OperatorSource CRs. See deploy/osd-curated-operators.

NOTE that ClusterVersion is being patched to add overrides. If other overrides are needed we'll have to tune how we do this patching. It must be done along with the OperatorSource patching to ensure CVO doesn't revert the OperatorSource patching.

Console Branding

In OSD, managed-cluster-config sets a key named branding to dedicated in the Console operator. This value is in turn read by code that applies the logo and other branding elements predefined for that value.

OAuth Templates

Docs TBA.

Resource Quotas

Refer to deploy/resource-quotas/README.md.

Image Pruning

Docs TBA.

Dependencies

pyyaml

Additional Scripts

There are additional scripts in this repo as a holding place for a better place or a better solution / process.

managed-cluster-config's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

managed-cluster-config's Issues

openshfit-aqua operatorgroup has malformed annotations

Hive is not able to apply the openshift-aqua OperatorGroup for the osd-aquasec-operator because the annotations are malformed.

See

olm.providedAPIs:
- AquaCsp.v1alpha1.operator.aquasec.com
- AquaDatabase.v1alpha1.operator.aquasec.com
- AquaEnforcer.v1alpha1.operator.aquasec.com
- AquaGateway.v1alpha1.operator.aquasec.com
- AquaScanner.v1alpha1.operator.aquasec.com
- AquaServer.v1alpha1.operator.aquasec.com

/cc @csheremeta

serviceAccount is deprecated -- use serviceAccountName

Placeholder to remind me to fix this.

diff --git a/deploy/efs-csi/04-daemonset.yaml b/deploy/efs-csi/04-daemonset.yaml
index e038271..79c26d1 100644
--- a/deploy/efs-csi/04-daemonset.yaml
+++ b/deploy/efs-csi/04-daemonset.yaml
@@ -16,7 +16,7 @@ spec:
         app: efs-csi-node
     spec:
       # DELTA: Added
-      serviceAccount: efs-csi-sa
+      serviceAccountName: efs-csi-sa
       # DELTA: Removed
       # priorityClassName: system-node-critical
       nodeSelector:

...but for all the things.

make fails

Make fails with permissions error:

[mjudeiki@localhost managed-cluster-config]$ make
rm -rf /home/mjudeiki/go1.11/src/github.com/openshift/managed-cluster-config/hack/00-osd-managed-cluster-config.selectorsyncset.yaml.tmpl
make[1]: Entering directory '/home/mjudeiki/go1.11/src/github.com/openshift/managed-cluster-config/deploy/resource-quotas'
make[1]: Leaving directory '/home/mjudeiki/go1.11/src/github.com/openshift/managed-cluster-config/deploy/resource-quotas'
make[1]: Entering directory '/home/mjudeiki/go1.11/src/github.com/openshift/managed-cluster-config/deploy/resource-quotas'
make[1]: Leaving directory '/home/mjudeiki/go1.11/src/github.com/openshift/managed-cluster-config/deploy/resource-quotas'
# The html goes into a secret.  if it's too big, it can't be updated so break it into one secret per html.
# Each SSS must not be too big as well.  each sub-dir of deploy/ becomes a SSS.  therefore each of the html
# becomes a separate dir.  This is a k8s limitation for annotation value size.
for TYPE in login providers errors; do \
        oc --config=.kubeconfig create secret generic osd-oauth-templates-$TYPE -n openshift-config --from-file=$TYPE.html=source/html/$TYPE.html --dry-run -o
 yaml > deploy/osd-oauth-templates-$TYPE/osd-oauth-templates-$TYPE.secret.yaml; \
done
if [ -z  ]; then \
        docker run --rm -v `pwd -P`:`pwd -P` python:2.7.15 /bin/sh -c "cd `pwd`; pip install oyaml; scripts/generate_syncset.py -t scripts/templates/ -y deplo
y -d /home/mjudeiki/go1.11/src/github.com/openshift/managed-cluster-config/hack/00-osd-managed-cluster-config.selectorsyncset.yaml.tmpl -r managed-cluster-con
fig"; \
else \
        scripts/generate_syncset.py -t scripts/templates/ -y deploy -d /home/mjudeiki/go1.11/src/github.com/openshift/managed-cluster-config/hack/00-osd-manag
ed-cluster-config.selectorsyncset.yaml.tmpl -r managed-cluster-config; \
fi
DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won't be maintained after that date. A f
uture version of pip will drop support for Python 2.7.
Collecting oyaml
  Downloading https://files.pythonhosted.org/packages/00/37/ec89398d3163f8f63d892328730e04b3a10927e3780af25baf1ec74f880f/oyaml-0.9-py2.py3-none-any.whl
Collecting pyyaml (from oyaml)
  Downloading https://files.pythonhosted.org/packages/e3/e8/b3212641ee2718d556df0f23f78de8303f068fe29cdaa7a91018849582fe/PyYAML-5.1.2.tar.gz (265kB)
Building wheels for collected packages: pyyaml
  Building wheel for pyyaml (setup.py): started
  Building wheel for pyyaml (setup.py): finished with status 'done'
  Stored in directory: /root/.cache/pip/wheels/d9/45/dd/65f0b38450c47cf7e5312883deb97d065e030c5cca0a365030
Successfully built pyyaml
Installing collected packages: pyyaml, oyaml
Successfully installed oyaml-0.9 pyyaml-5.1.2
You are using pip version 19.0.3, however version 19.3.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
/bin/sh: 1: scripts/generate_syncset.py: Permission denied
make: *** [Makefile:41: generate-syncset] Error 126
[mjudeiki@localhost managed-cluster-config]$ cat /etc/fedora-release 
Fedora release 30 (Thirty)

UpgradeConfig CR with LOCAL config does not sync.

Operator Image: quay.io/app-sre/managed-upgrade-operator:latest

Operator keeps reconciling but doesn't add status fields to the UpgradeConfig CR:

{"level":"info","ts":1626093892.7293298,"msg":"Using local CR as the upgrade config provider"}
{"level":"info","ts":1626093892.731769,"logger":"upgradeconfig-localprovider","msg":"Read the upgrade config from the cluster directly"}
{"level":"info","ts":1626094079.5829825,"logger":"controller_machineconfigpool","msg":"Reconciling MachineConfigPool","Request.Namespace":"","Request.Name":"worker"}
{"level":"info","ts":1626094174.783578,"msg":"Using local CR as the upgrade config provider"}
{"level":"info","ts":1626094174.78903,"logger":"upgradeconfig-localprovider","msg":"Read the upgrade config from the cluster directly"}
{"level":"info","ts":1626094375.0481997,"logger":"controller_machineconfigpool","msg":"Reconciling MachineConfigPool","Request.Namespace":"","Request.Name":"worker"}

UpgradeConfig CR:

apiVersion: upgrade.managed.openshift.io/v1alpha1
kind: UpgradeConfig
metadata:
  name: managed-upgrade-config-local
  namespace: openshift-managed-upgrade-operator
spec:
  type: "OSD"
  upgradeAt: "2021-07-06T10:00:00Z"
  PDBForceDrainTimeout: 60
  capacityReservation: true
  desired:
    channel: "stable-4.7"
    version: "4.7.18"

Config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: managed-upgrade-operator-config
  namespace: openshift-managed-upgrade-operator
data:
  config.yaml: |
    configManager:
      source: LOCAL
      localConfigName: managed-upgrade-config
      watchInterval: 5
    maintenance:
      controlPlaneTime: 90
      ignoredAlerts:
        controlPlaneCriticals:
        - ClusterOperatorDown
        - ClusterOperatorDegraded
    scale:
      timeOut: 30
    upgradeWindow:
      delayTrigger: 30
      timeOut: 120
    nodeDrain:
      timeOut: 45
      expectedNodeDrainTime: 8
    healthCheck:
      ignoredCriticals:
      - DNSErrors05MinSRE
      - MetricsClientSendFailingSRE
      - UpgradeNodeScalingFailedSRE
      - UpgradeClusterCheckFailedSRE
      - PrometheusRuleFailures
      - CannotRetrieveUpdates
      - FluentdNodeDown
      ignoredNamespaces:
      - openshift-logging
      - openshift-redhat-marketplace
      - openshift-operators
      - openshift-customer-monitoring
      - openshift-route-monitoring-operator
      - openshift-user-workload-monitoring
      - openshift-pipelines
    extDependencyAvailabilityChecks: {}
    verification:
      ignoredNamespaces:
      - openshift-logging
      namespacePrefixesToCheck:
      - openshift
      - kube
      - default

Makefile failing in CI workflow (release/prow)

A Dockerfile has been created to integrate this repo to the openshift/release process, so every time a PR is created, make will be executed inside a container, generating the assets and validating the process.

There seems to be a problem that only manifests in the CI platform, causing the following command to fail, apparently because git config does not return the expected result:

basename $$((git config --get-regex remote\.*\.url 2>/dev/null | cut -d ' ' -f2 || pwd) | head -n1 | sed 's|.git||g')

https://github.com/openshift/managed-cluster-config/blob/master/project.mk

Building the image locally with docker or imagebuilder (same tool used by ci-operator) works fine.

The failed CI job: https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/pr-logs/pull/openshift_release/4323/rehearse-4323-pull-ci-openshift-managed-cluster-config-master-images/9

@ALimaRedHat

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.