@saikyt Great suggestion, I'll add those notes to the docs.

from infra.

The server is failing to stand-up because the postgres deployment is failing, which it relies on. The error message indicates there are unbound persistent volume claims:

Warning FailedScheduling 17s (x9 over 40m) default-scheduler 0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.

This could be caused by a misconfiguration. Do you have a persistent volume defined? Troubleshooting this should get things moving.

from infra.

I do not have a PV volume defined. I was following the instructions from https://infrahq.com/docs/reference/self-hosting and did not see any step where I should be doing that. I can create a PV manually, but need some info, Is there a specific mount path I should be using? I am running int on a Ubuntu Host, should this be a host path? also, how much storage capacity should I allocate to Postgres? It would be great if you could share a sample YAML that you use. Thanks

from infra.

Hi @saikyt is this a managed cluster? If not, do you have a storage backend defined?

from infra.

@mxyng this is not a managed cluster, I have K8s running on a simple Ubuntu host.

from infra.

Do you have existing PVCs or are you able to create non-Infra PVCs? If so what storage class are they using? The default for Infra is to use the cluster default. If one doesn't exist, you can set it explicitly with postgres.persistence.className

from infra.

It seems the helm chart created a PVC but there is no corresponding PV mount for it.
Should the helm chart create the PV mount as well?

kubectl describe pvc
Name: infra-server-postgres
Namespace: default
StorageClass:
Status: Pending
Volume:
Labels: app.kubernetes.io/instance=infra-server
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=infra-server-postgres
app.kubernetes.io/version=0.21.0
helm.sh/chart=infra-server-0.1.2
Annotations: meta.helm.sh/release-name: infra-server
meta.helm.sh/release-namespace: default
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By: infra-server-postgres-0
Events:
Type Reason Age From Message

Normal FailedBinding 2m32s (x40347 over 7d) persistentvolume-controller no persistent volumes available for this claim and no storage class is set

from infra.

BTW there are no other PVCs.

from infra.

The Helm chart doesn't create a PV as it expects one to be created dynamically by the default, or configured, storage class. In EKS, this will look something like this

$ kubectl get storageclass
NAME            PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
gp2 (default)   kubernetes.io/aws-ebs   Delete          WaitForFirstConsumer   false                  400d

For an unmanaged cluster, you'll need to set a default storage class or configure it explicitly with the Helm value I mentioned earlier. If a storage class does not exist, you'll need to create one.

Alternatively, you can create a hostPath PV out of band and reference it using the same Helm values.

e.g.

# persistentvolume.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: infra-server-postgres
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  storageClassName: infra-server-postgres
  hostPath:
    path: /tmp/infra-server-postgres

# values.yaml
postgres:
  persistence:
    className: infra-server-postgres

Apply both configurations:

kubectl apply -f persistentvolume.yaml
helm install infra-server infrahq/infra-server -f values.yaml

A word of caution though. hostPath is intended for debugging/developing as it mounts a node local file/directory. If the pod gets scheduled on a different node, this persistent volume will no longer be available. An improvement on hostPath is local which requires configuring nodeAffinity and is not included in this example for simplicity.

from infra.

@mxyng That worked! thank you. However, I had to add the load balancer to the values.yaml before reInstalling with helm.

values.yaml

server:
service:
type: LoadBalancer

postgres:
persistence:
className: infra-server-postgres

Also, how do we address the root cause here? "The Helm chart doesn't create a PV as it expects one to be created dynamically by the default". Ideally, a local PV should be created dynamically. I don't know where and how that should happen. And, as you mentioned, hostPath is a temporary solution.

from infra.

Unfortunately PVs are a highly cluster-dependent configuration. What works for one cluster may not work for another. In the case of Infra server, it delegates configuration of the PV to the user. If a default StorageClass is defined, the deployment should succeed without issue. If one is not defined, as in your case, the PVC storage class will need to be configured explicitly, as the example above demonstrates. This storage class may then dynamically create the PV or, in your case, reference a PV that's been created statically. There's no good defaults for the latter since hostPath is susceptible to pod scheduling issues while all other volume types require additional configuration or cluster knowledge.

from infra.

Resolving this for now, issues with PVCs may be on a case-by-case basis as Mike notes.

Thanks for bringing this to our attention, please let us know if you face any further issues.

from infra.

Thank you @BruceMacD. Can we update the documentation that a default StorageClass should be defined for dynamic provisioning? It was not clear form the documentation. Also, when we define local storage class that does not support dynamic provisioning so in some cases user will have to create a static PV.

from infra.