Mention cilium
https://github.com/SUSE/avant-garde/issues/120

https://github.com/SUSE/avant-garde/issues/25

From rocket.chat:
https://chat.suse.de/channel/caasp-devel?msg=ZbwH5tYsrzjGQ4q8W

Change /etc/sysconfig/crio

cat /etc/sysconfig/crio 
NO_PROXY="localhost,127.0.0.1,3.0.0.0/8,192.168.0.0/16,10.0.0.0/8,.ge.com"
HTTP_PROXY="http://3.28.29.241:88/"
HTTPS_PROXY="http://3.28.29.241:88/"

Then restart with systemctl restart crio

4.4.3 Automatic Installation Using AutoYaST

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_deployment/book_caasp_deployment.html#sec.deploy.nodes.worker_install.manual.autoyast

The network setup in this section is confusing because it calls for netsetup for DHCP, but then points to the Advanced Network Setup section of Autoyast, which only references ifcfg.

ifcfg has more functionality and is preferred in the Autoyast documentation, so all references to netsetup should be replaced with ifcfg.

In addition, an example of the entire boot option line, based on using a static IP, should be given. For instance:
"For a static IP assignment, the Boot Option line will accept this syntax:
autoyast=http:///autoyast ifcfg=eth0=/,,, hostname=
For example:
autoyast=http://admin.my-caasp.com/autoyast ifcfg=eth0=192.168.100.11/24,192.168.100.1,192.168.100.2,my-caasp.com hostname=master1.my-caasp.com"

https://github.com/SUSE/avant-garde/issues/16

Be more verbose between step 1 and 2 since the salt-master can take some time to start. Also there is no word on how much time to wait between step 2 and 3.

Inform that step 2 might take several minutes so the user doesn't close the terminal before it finishes (there is no output for quite some time, no information about what is being done),

3.1.4 Recovering From Failed Updates

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#sec.admin.software.transactional-updates.recovering

Move the quickstart guide from Confluence to GitHub.

https://confluence.suse.de/display/SUSECaaSPlatform4/SUSE-CaaSP+vNext+Quickstart+Guide

• Translate Confluence page to AsciiDoc
• Configure preview build on susedoc.github.io
• Publish on susedoc.github.io

• Update release notes with new version
• Update entity for kubedocs

Procedure 7.2: Creating a Pod with RBD in Persistent Volume

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#id-1.5.9.5.3

I was just walking through this section and setting up Persistent Volumes in CaaSP in my lab and as written with the yaml that includes:
monitors:
- [MONITOR1_IP:MONITOR1_PORT]
- [MONITOR2_IP:MONITOR2_PORT]
- [MONITOR3_IP:MONITOR3_PORT]

I removed 2 of the MONITOR entries and used this:
monitors:
- [MONITOR1_IP:MONITOR1_PORT]

And when I tried to apply that yaml I got this error:
kubectl apply -f ceph-create-persistent-vol.yaml
error: error converting YAML to JSON: yaml: line 12: found unexpected ':'

I changed the brackets to single quotes and it worked:
monitors:
- '192.168.0.184:6789'

kubectl apply -f ceph-create-persistent-vol.yaml
persistentvolume "bradpv" created

I assume something has changed in kubernetes that the brackets no longer work?

4.4.3 Automatic Installation Using AutoYaST

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_deployment/book_caasp_deployment.html#sec.deploy.nodes.worker_install.manual.autoyast

In this section it says "However, each node will have ssh keys for root on the master node pre-installed."

This is not true. The keys are on the Administration node, not on any of the K8s master nodes.

We should have a suggestion note that nodes should not be overloaded (or even committed) so workloads can have more chances from moving from one node to another during an update or maintenance window.

[email protected] @ [email protected] on 2019-03-25:

In the Administration Guide, we recommand create kubernetes-dashboard
in the namespace of "kube-system", but missed in the "Get the
Kubernetes Dashboard URL by running" session.

So please add namespace to dashboad by changing

export NODE_PORT=$(kubectl get -o jsonpath="{.spec.ports[0].nodePort}" services kubernetes-dashboard)

to

export NODE_PORT=$(kubectl get -o jsonpath="{.spec.ports[0].nodePort}" services kubernetes-dashboard --namespace=kube-system)

Procedure 3.6: Installation of Kubernetes Dashboard

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#id-1.5.5.6.4

Since section 3.4.3 covers installing heapster, step one in procedure 3.6 should have a note that this step can be skipped if heapster is already installed.

Acceptance Criteria

• Describe deployment scenario
• Which platform
• How many nodes
• How are the nodes meant to interact
• Describe how to deploy and configure vNext nodes on a SLE level with the preferred method
• Describe which packages to install to prepare for CaaSP deployment
• Describe how to configure networking and what needs to be considered
• Describe how to deploy CaaSP vNext software to the nodes
• Describe how to bootstrap the cluster

https://confluence.suse.de/display/SUSECaaSPlatform4/Quick+start+guide+on+deploying+vNext

https://github.com/r0ckarong/caasp-rfc
https://github.com/SUSE/avant-garde/issues/150

There is no such thing as "openSUSE 15", but it's still mentioned in various places.

2.2.7.2 Client Configuration

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_deployment/book_caasp_deployment.html#sec.deploy.scenarios.airgap.helm-charts.client

--> the name of the helm-mirror is case-sensitive and it seems that only "lower case" works.

Please change "SUSE-MIRROR" to "suse-mirror" as otherwise the fetch will not work.

https://github.com/SUSE/doc-caasp/releases/tag/v3-20190409

https://bugzilla.suse.com/show_bug.cgi?id=1131953

8.5 Locking Installed Program Temporary Fixes

http://docserv.suse.de/documents/CaaS_Platform_3/caasp-admin/single-html/#sec.admin.troubleshooting.velum_ptf

There is a problem that the comand to install PTF do not work:
On run of:

1. Install the PTF manually.
   root: transactional-update reboot PTF_FILE.rpm install

I do get
transactional-update reboot *.rpm install
Usage: transactional-update --help|--version
transactional-update [--no-selfupdate] [cleanup][up|dup|patch|initrd][kdump][reboot]
transactional-update [--no-selfupdate] [cleanup] [reboot] pkg install|remove|update PKG1..PKGN
transactional-update [--no-selfupdate] migration
transactional-update [--no-selfupdate] register -p [-r ]
transactional-update rollback [number]

But I can run:
transactional-update pkg install TESTrpm

So this have to be changed in my opinion.

transactional-update pkg install PTFrpm

1 Security
https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#cha.admin.security

The configuration of the OIDC Provider is missing from the documentation.

Procedure 2.9: Pull Data From Upstream Sources

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_deployment/book_caasp_deployment.html#id-1.4.4.3.12.4

When I follow these instructions I get this output of helm-mirror:

helm-mirror inspect-images /srv/www/htdocs/kubernetes-charts.suse.com/ -o skopeo=x.yaml
helm-mirror: 2019/02/05 14:57:55.037049 inspectImages.go:136: error: cannot render chart: render error in "cf/templates/tcp-router.yaml": template: cf/templates/tcp-router.yaml:42:28: executing "cf/templates/tcp-router.yaml" at <include (print $.Tem...>: error calling include: template: cf/templates/secrets.yaml:5:29: executing "cf/templates/secrets.yaml" at <required "secrets.CL...>: error calling required: secrets.CLUSTER_ADMIN_PASSWORD has not been set
helm-mirror: 2019/02/05 14:57:55.037206 inspectImages.go:93: error: cannot load chart: render error in "cf/templates/tcp-router.yaml": template: cf/templates/tcp-router.yaml:42:28: executing "cf/templates/tcp-router.yaml" at <include (print $.Tem...>: error calling include: template: cf/templates/secrets.yaml:5:29: executing "cf/templates/secrets.yaml" at <required "secrets.CL...>: error calling required: secrets.CLUSTER_ADMIN_PASSWORD has not been set
helm-mirror: 2019/02/05 14:57:55.037251 inspectImages.go:100: error walking the path "/srv/www/htdocs/kubernetes-charts.suse.com/": render error in "cf/templates/tcp-router.yaml": template: cf/templates/tcp-router.yaml:42:28: executing "cf/templates/tcp-router.yaml" at <include (print $.Tem...>: error calling include: template: cf/templates/secrets.yaml:5:29: executing "cf/templates/secrets.yaml" at <required "secrets.CL...>: error calling required: secrets.CLUSTER_ADMIN_PASSWORD has not been set
helm-mirror: 2019/02/05 14:57:55.037285 inspectImages.go:62: error: procesing target /srv/www/htdocs/kubernetes-charts.suse.com/: render error in "cf/templates/tcp-router.yaml": template: cf/templates/tcp-router.yaml:42:28: executing "cf/templates/tcp-router.yaml" at <include (print $.Tem...>: error calling include: template: cf/templates/secrets.yaml:5:29: executing "cf/templates/secrets.yaml" at <required "secrets.CL...>: error calling required: secrets.CLUSTER_ADMIN_PASSWORD has not been set
Error: render error in "cf/templates/tcp-router.yaml": template: cf/templates/tcp-router.yaml:42:28: executing "cf/templates/tcp-router.yaml" at <include (print $.Tem...>: error calling include: template: cf/templates/secrets.yaml:5:29: executing "cf/templates/secrets.yaml" at <required "secrets.CL...>: error calling required: secrets.CLUSTER_ADMIN_PASSWORD has not been set
Usage:
mirror inspect-images [folder|tgzfile] [flags]

Flags:
-h, --help help for inspect-images
-o, --output string choose an output for the list of images and specify
the file name, if not specified 'images.out' will be the default.
Options:

                    - file: outputs all images to a file
                    - json: outputs all images to a file in JSON format
                    - skopeo: outputs all images to a file in YAML format
                      to be used as source file with the 'skopeo sync' command.
                      For more information refer to the 'skopeo sync'
                      documentation at https://github.com/SUSE/skopeo/blob/sync/docs/skopeo.1.md#skopeo-sync
                    - stdout: prints all images to standard output
                    - yaml: outputs all images to a file in YAML format

                    Usage:

                            - helm mirror inspect-images /tmp/helm --output stdout
                            - helm mirror inspect-images /tmp/helm -o stdout
                            - helm mirror inspect-images /tmp/helm -o file=filename
                            - helm mirror inspect-images /tmp/helm -o json=filename.json
                            - helm mirror inspect-images /tmp/helm -o yaml=filename.yaml
                            - helm mirror inspect-images /tmp/helm -o skopeo=filename.yaml

                     (default "stdout")

Global Flags:
-i, --ignore-errors ignores errors while downloading or processing charts
-v, --verbose verbose output

render error in "cf/templates/tcp-router.yaml": template: cf/templates/tcp-router.yaml:42:28: executing "cf/templates/tcp-router.yaml" at <include (print $.Tem...>: error calling include: template: cf/templates/secrets.yaml:5:29: executing "cf/templates/secrets.yaml" at <required "secrets.CL...>: error calling required: secrets.CLUSTER_ADMIN_PASSWORD has not been set

https://github.com/SUSE/avant-garde/issues/132

Type: Bug

Topic: Resource reservations with velum

Impact: big if out of resources / low as long enough resources available

Description
The resource reservations set in velum are not properly applied to the cluster nodes. The kubelet process does not have the correct parameters.

There is no impact at all as long the nodes have enough resources. But if the situation occurs where pods gather to many resources there is no possibility to analyze the problem because the overloaded node is not reacting anymore. Only chance is to power off the node.

Current behavior
Both, the kube core service and host system service reservations are applied to the kubelet process with --kube-reserved. See this example:

/usr/bin/hyperkube kubelet --logtostderr=true --v=2 --hostname-override=node1 --allow-privileged=true --config=/etc/kubernetes/kubelet-config.yaml --kube-reserved=cpu=500m,memory=1G --kube-reserved=cpu=100m,memory=512M --node-ip=10.10.10.10 --pod-infra-container-image=sles12/pause:1.0.0 --network-plugin=cni --cni-bin-dir=/var/lib/kubelet/cni/bin --cni-conf-dir=/etc/cni/net.d --kubeconfig=/var/lib/kubelet/kubelet-config --volume-plugin-dir=/usr/lib/kubernetes/kubelet-plugins

Expected behavior
The values entered for host system services should be applied with --system-reserved as shown in the example below:

/usr/bin/hyperkube kubelet [...] --kube-reserved=cpu=500m,memory=1G --system-reserved=cpu=100m,memory=512M [...]

Once air gap for v4 is developed we need to look at the input and applicability of:

https://bugzilla.suse.com/show_bug.cgi?id=1073507
https://bugzilla.suse.com/show_bug.cgi?id=1139748

Procedure 9.1: Installation of Kubernetes Dashboard

https://susedoc.github.io/doc-caasp/develop/caasp-user/single-html/#id-1.6.10.2.4

Gives a 404 when trying to run kubectl apply -f
https://raw.githubusercontent.com/kubernetes/dashboard/master/src/deploy/recommended/kubernetes-dashboard.yaml

https://bugzilla.suse.com/show_bug.cgi?id=1125349#c7

On the mapping side of the documentation I had to guess basically.

Additionally I believe we we need to clarify the
fact that the default DN in the membership attribute does not work. That you
only need the plain cn/uid in order for it to be mapped.

4.3 Configuring the Administration Node

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_deployment/book_caasp_deployment.html#sec.deploy.nodes.admin_configuration

I would recommend a better description of this setting. It took me a while to figure out what this would do, and Google won't help at all in this case.

https://bugzilla.suse.com/show_bug.cgi?id=1124651

if the customer is deploying CaaSP in public cloud then you can deploy using the loadbalancer type. If he deploys in BareMetal he has to provide an external load balancer

The documentation for portus on SLE is missing a lot of information.

For starters:

Portus is accessible for SLES customers as a Docker image.

This is missing the location where this Docker image can be pulled: the SUSE Container Registry. Portus 2.4 has been released there as a Docker image and it's based on SLE, so customers shouldn't be pulling from Dockerhub, ever. And they should not be looking at the containers module either.

For more information and documentation about Portus, see: http://port.us.org/docs/deploy.html.

This is true, but it's missing quite a lot of info. The difference between the community and the customer-supported version of the Docker image resides only on the base image and how we build said images (the community version is built on Dockerhub, the customer one on IBS). Besides that, there's no real difference: it's the same code. So, setting up the customer-supported version and maintaining it is exactly the same as doing so with the community image. This part of the documentation should be more clear on this because otherwise, customers can get lost.

Moreover, as explained on this page, we also provide some examples on how to deploy Portus here. Maybe it would be worth it to also write this here.

2.5 Scaling the Cluster

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#sec.admin.scale_cluster

Incorrect:

The default maximum number of nodes in a cluster is 40. The Salt Master configuration needs to be adjusted to handle installation and updating a of larger cluster:

Correct

The default maximum number of nodes in a cluster is 150. The Salt Master configuration needs to be adjusted to handle installation and updating a of larger cluster:

https://github.com/SUSE/avant-garde/issues/115

Once the product is shipped as an extension this needs to be discussed in release notes.

https://github.com/SUSE/avant-garde/issues/97

SUSE/caasp-salt#724 (comment)

Procedure 1.2: Changing Velum Admin Password

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#id5412

docker exec -it $(docker ps -q -f name=ldap) /bin/bash

bash-4.3# slappasswd -n -s password
{SSHA}SSK3rBatmgUNrDmohlOw5JMhVB/vvEcG

bash-4.3# ldappasswd -H ldaps:// -D "cn=admin,dc=infra,dc=caasp,dc=local" \
-w $(cat /var/lib/misc/infra-secrets/openldap-password) \
uid=test,ou=People,dc=infra,dc=caasp,dc=local -s {SSHA}SSK3rBatmgUNrDmohlOw5JMhVB/vvEcG
Result: No such object (32)

You can try to run this command by replacing the target with the minion-id.

docker exec -ti $(docker ps -q --filter name=salt-master) salt-run
--force-color state.orchestrate orch.removal
pillar='{"target":"87019cb873604c21b94b0f3d91d96369"}'

4.2.2.1 kubelet

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#sec.admin.monitoring.health.node.kubelet

curl -i https://localhost:10250/healthz doesn't work anymore

curl -i https://master01.infra.caasp.local:10250/healthz
HTTP/1.1 401 Unauthorized
Date: Thu, 21 Mar 2019 10:33:36 GMT
Content-Length: 12
Content-Type: text/plain; charset=utf-8

Unauthorized

It seems kubelet is now configured with port 10248 (HTTP endpoint).

curl -i http://localhost:10248/healthz
HTTP/1.1 200 OK
Date: Thu, 21 Mar 2019 10:36:24 GMT
Content-Length: 2
Content-Type: text/plain; charset=utf-8

ok

Nowhere in this document, the registration process is described. It is possible (to what effect I don't know) to use SUSEConnect or transactional-update to register a system against SCC. The correct order the cluster has to be registered and update is also not found here.

3.1.3 Applying Updates

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#sec.admin.software.transactional-updates.installation

Discussed in Core Platform call 2019-04-09. RBAC documentation is poor. Upstream documentation is confusing and not helpful. Could be an opportunity to set apart CaaS Platform with good docs.

Alternatively: Fix upstream docs.

Need more input for actual use cases and what needs to be described.
Needs prioritization from CaaSP mgmt to get engineers to describe it and develop test cases.

https://confluence.suse.de/display/SUSECaaSPlatform4/v4?focusedCommentId=199295492#comment-199295492

Needs to clarify if internal or external LDAP is supported and how to configure/use

• https://github.com/SUSE/avant-garde/issues/164
• https://github.com/SUSE/avant-garde/issues/19
• https://github.com/SUSE/avant-garde/issues/17

Output reviews with:

• #298
• #292
• #293

3.1.4 Recovering From Failed Updates

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_admin/book_caasp_admin.html#sec.admin.software.transactional-updates.recovering

We should target only the real minions using

salt -P "roles:(admin|kube-(master|minion))"

otherwise we will target also the ca container.

topic: wrong LDAP TLS port

source: https://github.com/SUSE/doc-caasp/blob/develop/xml/admin_security.xml#L1407

correction: should be 636 instead of 646

5.2.5.11 Assigning Roles to the Cluster Nodes

https://www.suse.com/documentation/suse-caasp-3/singlehtml/book_caasp_deployment/book_caasp_deployment.html#sec.deploy.cloud-init.user-data.caasp-roles

With CaaSP v4, the cluster role does no longer configure a NTP client. Instead the admin has to configure one (either chrony or systemd-timesyncd) with cloud-init himself. NTP syncronisaton across the cluster is required.

The following text needs to be adjusted:
"...and configure a timesync service with administration node as a reference. You do not have to install any NTP server, but if you need to use one, you need to disable the systemd-timesyncd first."

3.1.4 Recovering From Failed Updates

http://docserv.suse.de/documents/CaaS_Platform_3/caasp-admin/single-html/#sec.admin.software.transactional-updates.recovering

On run of documented 2nd step:

docker exec -it $(docker ps -q -f name="salt-master")
salt "*" cmd.run "transactional-update cleanup reboot dup

I do get on my CaaSP 3.0 set up:
"docker: command not found"

Relates to: #146

We describe adding new nodes into the cluster as replacements for failed
ones. These nodes get installed from installation media. I am unsure if
they update from release/update channels during the update.

It depends on how the nodes are installed.

Media installation

The user installs the node using the v3 installation media. The node
will get all the updated packages only if the customer register it
against SMT/SCC/RMT at installation time.

AutoYaST

We have customers installing nodes using hand-made autoyast profiles.
This is something we don't support. So let's assume they stick to the
supported way and they use the profile we generate on the admin node.

If the admin node is registered against SMT/SCC/RMT then nodes will be
bootstrapped using updates.

Pre-build virtual image

The customer installs everything using a prebuilt qcow image for
example. In this case the node won't have the updates applied.

The only way we officially describe to update nodes is to do that
through velum. Is this a problem at all if we have some "old" nodes with
newer packages? Will Velum manage to bring the new nodes with the
outdated packages up to the same state as the ones from the old cluster?

The customer can install all the updates manually before joining the
node to the cluster. If he doesn't do that he will have to update the
node using Velum's UI.

I would strongly encourage them to either install a node with all the
updates already applied or to update them before assigning a role to
the node. That's because of the usual reboot process and how many things
can go south during that.

Describe VMWare deployment

https://github.com/SUSE/avant-garde/issues/131

https://jira.suse.de/browse/CA-471

https://github.com/SUSE/avant-garde/issues/15