ryanhay / ocp4-metal-install Goto Github PK

When issueing this command

setsebool -P haproxy_connect_any 1 # SELinux name_bind access

I get the following error and cannot enable haproxy

[root@ocp-svc haproxy]# libsepol.context_from_record: type hwtracing_device_t is not defined
libsepol.context_from_record: could not create context structure
libsepol.context_from_string: could not create context structure
libsepol.sepol_context_to_sid: could not convert system_u:object_r:hwtracing_device_t:s0 to sid
invalid context system_u:object_r:hwtracing_device_t:s0
Failed to commit changes to booleans: Success

RHEL 9

Hello Sir,

I do the same configuration of Service node on CentOS 7 rather than 8 because I was facing issues to download repository on CentOS8. All configuration I done all was successful on CentOS 7 but whenever I try to pull the files on Bootstrap node it failed to download the files from the URL

I attached the screenshot of error

Hi,
I´m using the guide HaProxy config,and when enabled the HA service it shows this error.

Jun 28 20:09:05 ocp-svc.ocp.lan systemd[1]: Starting HAProxy Load Balancer...
Jun 28 20:09:05 ocp-svc.ocp.lan haproxy[2705]: [ALERT] (2705) : parsing [/etc/haproxy/haproxy.cfg:92]: Missing LF on last line, file might have been truncated at position >
Jun 28 20:09:05 ocp-svc.ocp.lan haproxy[2705]: [ALERT] (2705) : Error(s) found in configuration file : /etc/haproxy/haproxy.cfg
Jun 28 20:09:05 ocp-svc.ocp.lan haproxy[2705]: [ALERT] (2705) : Fatal errors found in configuration.
Jun 28 20:09:05 ocp-svc.ocp.lan systemd[1]: haproxy.service: Control process exited, code=exited, status=1/FAILURE
Jun 28 20:09:05 ocp-svc.ocp.lan systemd[1]: haproxy.service: Failed with result 'exit-code'.
Jun 28 20:09:05 ocp-svc.ocp.lan systemd[1]: Failed to start HAProxy Load Balancer.

Hi,

Following your install guide got an error on worker node installation with attempt more then 600 times and keep going. See attach file for the screenshot.

Please help on what needs to be check and done here.

Step 1 - I can not get either option to work.

Q1: Is rhcos equal to the rhcos*.iso file?
Q2: For the bootstrap method, do you replace the entire line after pressing the TAB?
Q3: For the other method I get the following error:

# Bootstrap Node - ocp-bootstrap
coreos.inst.install_dev=sda coreos.inst.image_url=http://192.168.22.1:8080/ocp4/rhcos coreos.inst.insecure=yes coreos.inst.ignition_url=http://192.168.22.1:8080/ocp4/bootstrap.ign

# Or if you waited for it boot, use the following command then just reboot after it finishes and make sure you remove the attached .iso
sudo coreos-installer install /dev/sda -u http://192.168.22.1:8080/ocp4/rhcos -I http://192.168.22.1:8080/ocp4/bootstrap.ign --insecure --insecure-ignition

Hey Ryan,

I have a couple of suggestions for your NFS section (18).

1. create a couple of extra shares for when the cluster is ready - folks are going to need a few, might as well set them up here...

   mkdir -p /shares/{registry,pv0001,pv0002,pv0003}
   chown -R nobody:nobody /shares/{registry,pv0001,pv0002,pv0003}
   chmod -R 777 /shares/{registry,pv0001,pv0002,pv0003}

and

echo "/shares/registry 192.168.22.0/24(rw,sync,root_squash,no_subtree_check,no_wdelay)" > /etc/exports
echo "/shares/pv0001 192.168.22.0/24(rw,sync,root_squash,no_subtree_check,no_wdelay)" >> /etc/exports
echo "/shares/pv0002 192.168.22.0/24(rw,sync,root_squash,no_subtree_check,no_wdelay)" >> /etc/exports
echo "/shares/pv0003 192.168.22.0/24(rw,sync,root_squash,no_subtree_check,no_wdelay)" >> /etc/exports
exportfs -arv

2. Folks should pay close attention to the subnet and use the correct one. I deployed a cluster on IBM's cloud and my subnet was 10.70.174.128/26 - which looks like a normal IP address but isn't, it's a "network" because the CIDR is /26 ;-)

I would even recommend spinning up an extra "test" vm on the cluster's subnet and actually testing the NFS mounts before attempt to use them persistent volumes:

sudo mkdir /test
mount -t nfs ocp-svc:/shares/registry /test
touch /test/it-works
rm /test/it-works
umount /test

If this works, it can save you some real issues down the road...

You could also use the VM to validate some network routes and name resolution to ensure the firewall is has opened it's ports and is forwarding traffic correctly.

Thanks again for your great work capturing all of this.

E.

Looks like the openshift env built, but not able to access the following I see:

https://192.168.2.200:6443/

{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "forbidden: User "system:anonymous" cannot get path "/"",
"reason": "Forbidden",
"details": {

},
"code": 403
}

https://192.168.2.200:9000/stats

This site can’t provide a secure connection192.168.2.200 sent an invalid response.
ERR_SSL_PROTOCOL_ERROR

Details of configured env below.

[root@okd4-services ~]# curl -kv https://oauth-openshift.apps.lab.ocp.lan/healthz

• Trying 192.168.2.200...
• TCP_NODELAY set
• Connected to oauth-openshift.apps.lab.ocp.lan (192.168.2.200) port 443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• successfully set certificate verify locations:
• CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to oauth-openshift.apps.lab.ocp.lan:443
  curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to oauth-openshift.apps.lab.ocp.lan:443

[root@okd4-services ~]# sh -x /tmp/.g

• oc get csr
  No resources found
• oc get nodes
  NAME STATUS ROLES AGE VERSION
  okd4-compute-1.lab.ocp.lan Ready worker 6h10m v1.22.0-rc.0+a44d0f0
  okd4-compute-2.lab.ocp.lan Ready worker 5h35m v1.22.0-rc.0+a44d0f0
  okd4-control-plane-1.lab.ocp.lan Ready master,worker 6h20m v1.22.0-rc.0+a44d0f0
  okd4-control-plane-2.lab.ocp.lan Ready master,worker 6h17m v1.22.0-rc.0+a44d0f0
  okd4-control-plane-3.lab.ocp.lan Ready master,worker 6h14m v1.22.0-rc.0+a44d0f0
• oc get pods
  NAME READY STATUS RESTARTS AGE
  myapache-7bcf9c6d44-tjmzs 1/1 Running 0 4h5m
  myapache-7bcf9c6d44-xmtqx 1/1 Running 0 4h5m
• oc get pods -n openshift-network-operator
  NAME READY STATUS RESTARTS AGE
  network-operator-6f4564ffb-b82hr 1/1 Running 15 (14m ago) 6h25m
• oc get daemonsets -n openshift-sdn
  NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
  sdn 5 5 5 5 5 kubernetes.io/os=linux 6h19m
  sdn-controller 3 3 3 3 3 node-role.kubernetes.io/master= 6h19m
• oc get network.config.openshift.io cluster -o yaml
  apiVersion: config.openshift.io/v1
  kind: Network
  metadata:
  creationTimestamp: "2021-11-12T17:50:19Z"
  generation: 2
  name: cluster
  resourceVersion: "3114"
  uid: 67438d09-d9df-4b9c-b19d-9863e990df05
  spec:
  clusterNetwork:
  • cidr: 10.128.0.0/14
    hostPrefix: 23
    externalIP:
    policy: {}
    networkType: OpenShiftSDN
    serviceNetwork:
  • 172.30.0.0/16
    status:
    clusterNetwork:
  • cidr: 10.128.0.0/14
    hostPrefix: 23
    clusterNetworkMTU: 1450
    networkType: OpenShiftSDN
    serviceNetwork:
  • 172.30.0.0/16

[root@okd4-services ~]# dig ocp.lan

; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> ocp.lan
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58296
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 0478592c3088da21edc7206c618f046c2b6a75fe33e2ea60 (good)
;; QUESTION SECTION:
;ocp.lan. IN A

;; AUTHORITY SECTION:
ocp.lan. 604800 IN SOA okd4-services.ocp.lan. contact.ocp.lan.ocp.lan. 1 604800 86400 2419200 604800

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Nov 12 19:18:52 EST 2021
;; MSG SIZE rcvd: 130

[root@okd4-services ~]# dig -x 192.168.22.211

; <<>> DiG 9.11.26-RedHat-9.11.26-4.el8_4 <<>> -x 192.168.22.211
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16721
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 1, ADDITIONAL: 2

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 4c8bb19fd94b10e384c6db3b618f046f084882a1da69e033 (good)
;; QUESTION SECTION:
;211.22.168.192.in-addr.arpa. IN PTR

;; ANSWER SECTION:
211.22.168.192.in-addr.arpa. 604800 IN PTR okd4-compute-1.lab.ocp.lan.

;; AUTHORITY SECTION:
22.168.192.in-addr.arpa. 604800 IN NS okd4-services.ocp.lan.

;; ADDITIONAL SECTION:
okd4-services.ocp.lan. 604800 IN A 192.168.22.1

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Nov 12 19:18:55 EST 2021
;; MSG SIZE rcvd: 168

For this step
cp -R ~/ocp-install/* /var/www/html/ocp4

This would also copy the ~/ocp-install/auth directory and exposed via the web server.
It would include the two files:
kubeadmin-password and kubeconfig

In case it is necessary for installation of control pane / nodes, would it be ok to remove the directory /var/www/html/ocp4/auth after installation?

I added two more workers to the cluster.
added their entries in HAProxy configuration.

Entries are listing on stats page but only 2 out of 4 workers details get updated on stats page. WHat more configuration I should do to see updates from all 4 worker nodes.

I'm trying to install openshift 4.9 using IPI and UPI but the same error appears in the bootstrap machine "overlayfs: unrecognized mount option ""volatile" or missing value"
and the installation didn't complete with the following output: "if anyone help i'll very thankful"

time="2021-11-08T05:38:16+02:00" level=error msg="Bootstrap failed to complete: timed out waiting for the condition"
time="2021-11-08T05:38:16+02:00" level=error msg="Failed to wait for bootstrapping to complete. This error usually happens when there is a problem with control plane hosts that prevents the control plane operators from creating the control plane."
time="2021-11-08T05:38:16+02:00" level=fatal msg="Bootstrap failed to complete"

======

I'm using vsphere 6.7U2
.openshift_install.log
.openshift-install.log is attached

hello, I have a problem when installing openshift 4.11 get the following error on the bootstrap machine.
Error: couldn't find boot device for /dev/sda
Resetting partition table
Error : install failed

https://photos.app.goo.gl/jFRcjr24hm11rVdN7

Hi ,

Your Guide was very useful,
just one question though , how to access the OpenShift image registry from outside the network, the image registry url ends with apps.lab.ocp.lan but when I try to access it from the ocp-svc machine , it doesn't show up, the pod is running and it has an internal IP (assigned by libvirt I think) , so how can I access it from the ocp-svc machine ?

Why do you use opc-svc.opc.lan and sometimes use ocp-svc.lab.ocp.lan?

"git clone https://github.com/ryanhay/ocp4-metal-install"

requires username and password when trying to run above command.

What we have to use to run this command?

This is more of a question rather than an issue @ryanhay . I would like to know if i can use a combination of ESXi and bare metals? I can spin up all the control plane on bare metal and even worker nodes on bare metals and keep services and bootstrap on the ESXi server?

My catch here is that the OCP network(2nd network that we create) how do i bridge that to my VMs? Also my bootstrap machine which is on ESXi is on the OCP network. But unable to fetch files from the Centos VM. Im a newbie to this, so these might sound like basic questions. Any help would be appreciated.

[root@ocp-svc ~]# ~/openshift-install --dir ~/ocp-install wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.9.18
DEBUG Built from commit eb132dae953888e736c382f1176c799c0e1aa49e
INFO Waiting up to 20m0s for the Kubernetes API at https://api.lab.ocp.lan:6443...
DEBUG Still waiting for the Kubernetes API: the server has asked for the client to provide credentials

can any one know why it's happening like this....

OS Suggestion For This Now that CentOS 8 is EOL? Thanks! Can Stream to used, or CentOS 7 is still in life, or RHEL?

Hello,

can you please advise?

Hi Ryan,

Thank you so much with this repository. I am running openshift and am trying to get the Bootstrap to complete and debugging it using the command ~/openshift-install --dir ~/ocp-install wait-for bootstrap-complete --log-level=debug.

Error:
INFO Waiting up to 20m0s for the Kubernetes API at https://api.lab.ocp.lan:6443/...
DEBUG Still waiting for the Kubernetes Get " https://api.lab.ocp.lan:6443/version":EOF.

If anyone else had the same error or knows how to fix it let me know.
Thanks

I deployed a oracle Database container and trying to access the DB using service. But after exposing the service, I'm not able to access or ping the external IP. i used nodeport and external IP as well but still not able to access the database.

[root@ocp-svc ~]# oc get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 172.30.0.1 443/TCP 21d
openshift ExternalName kubernetes.default.svc.cluster.local 21d
os-sample-java-web ClusterIP 172.30.104.87 8080/TCP,8443/TCP,8778/TCP 4d19h
sdsqa-oracle-svc02 NodePort 172.30.111.211 10.221.92.160 1521:31864/TCP 5d23h
sdsqa-oracle-svc03 NodePort 172.30.167.211 10.221.92.160 1521:31865/TCP 4d21h
sdsqa-oracle-svc04 NodePort 172.30.213.22 1521:31866/TCP 2d4h
[root@ocp-svc ~]#

Any idea how we can access applications using external IPs or nodeport mechanism.

I don't know if this was introduced with Openshift 4.7 ... but you need to change the kernel boot arguments otherwise it will fail to boot with missing gpg sig files that you can not get hold of ...

The official documents state ...

"If you are using coreos.inst.image_url, you must also use coreos.inst.insecure. This is because the bare-metal media are not GPG-signed for OpenShift Container Platform."

Your kernal boot documentation is omitting the final

coreos.inst.insecure

!!!
Took me 2 weeks to figure out ... I kept trying to sign the image files with the ssh-keygen which is impossible currently with the tools in RHEL 8.

Hello All
I am new to openshift, I have successfully deployed a cluster as per the all given instructions.
In Overview tab, Cluster status is "Ready", I am unable to open link through "Open Console" button.
Giving an error "This site can't be reached".

Attached snap for reference, Thanks in advance!!

Dears,
Kindly I need your help to download cluster and its component as the interface has a major change and cannot found the most of components
Thanks

Version
$ openshift-install version
./openshift-install 4.5.9
built from commit 0d5c871ce7d03f3d03ab4371dc39916a5415cf5c
release image quay.io/openshift-release-dev/ocp-release@sha256:7ad540594e2a667300dd2584fe2ede2c1a0b814ee6a62f60809d87ab564f4425
Platform:
baremetal

UPI (semi-manual installation on customised infrastructure)
What happened?
Cluster details:
Control planes on the baremetal and worker on baremetal. But the bootstrap is running on the ESXi server which is on the same network.

After i launch my boot strap and control nodes i can see this message for boot strap:

~/openshift-install --dir ~/ocp-install wait-for bootstrap-complete --log-level=debug
DEBUG OpenShift Installer 4.5.9
DEBUG Built from commit 0d5c871
INFO Waiting up to 20m0s for the Kubernetes API at https://api.lab.ocp.lan:6443...
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF
DEBUG Still waiting for the Kubernetes API: Get https://api.lab.ocp.lan:6443/version?timeout=32s: EOF

And on the Control plane nodes i can see the error as seen below image:

May i know if i have missed something? i fell some issue with connectivity.

Hi Ryan

I would suggest making it clear that the:

sed -i 's/mastersSchedulable: true/mastersSchedulable: false/' ~/ocp-install/manifests/cluster-scheduler-02-config.yml

command is not optional. I missed it the first time and the following instructions only work when the commands were run before. Great guide though, highly appreciated!

Hi Ryan,
thanks for the information here.
just one thing ... you generated the ssh key (ssh-keygen) using the root user on the svc node , but should it be generated by 'root' user or 'core' user ? as there is no root user on core OS

You may want to put a warning up on your docs :-) #2 months of pain.

https://bugzilla.redhat.com/show_bug.cgi?id=1935539

RH have issued a warning on their web site

Virtual machines (VMs) configured to use virtual hardware version 14 or greater might result in a failed installation. It is recommended to configure VMs with virtual hardware version 13. This is a known issue that is being addressed in BZ#1935539.

I have seen this with both ESXi 7.0b and Proxmox 6.3-6 ..may save people 2 months of pain I have been going through.

The nature of the problem means you just have to keep on restarting the bootstrap phase / install phase repeatedly ever time it times out and fails ... eventually, if lucky you will get through the problem within 24hrs.

This file (rhcos-X.X.X-x86_64-metal.x86_64.raw.gz) is no longer an available download when creating a cluster. Any idea what I should use instead?

I followed same steps to configure OCP4-Metal-install and I successfully configured the cluster with 3 control plane nodes and two worker nodes.
I added additional node to the cluster successfully but the status in not updating in HA-proxy. any changes do I need to perform in HA-Proxy. any support is much appreciated.

[root@ocp-svc log]# oc get nodes
NAME STATUS ROLES AGE VERSION
ocp-cp-1.lab.ocp.lan Ready control-plane,master,worker 28h v1.27.8+4fab27b
ocp-cp-2.lab.ocp.lan Ready control-plane,master,worker 27h v1.27.8+4fab27b
ocp-cp-3.lab.ocp.lan Ready control-plane,master,worker 27h v1.27.8+4fab27b
ocp-w-1.lab.ocp.lan Ready worker 26h v1.27.8+4fab27b
ocp-w-2.lab.ocp.lan Ready worker 26h v1.27.8+4fab27b
ocp-w-3.lab.ocp.lan Ready worker 7h37m v1.27.8+4fab27b
ocp-w-4.lab.ocp.lan Ready worker 7h17m v1.27.8+4fab27b
[root@ocp-svc log]#
[root@ocp-svc log]#
[root@ocp-svc log]#