Giter Club home page Giter Club logo

Comments (16)

mulbc avatar mulbc commented on June 24, 2024 1

Fixed @jgato 's issue in an interactive session - the problem was that he created the LVMCluster CR in the default namespace instead of the lvm-operator-system namespace (the sample yaml does not specify a namespace)

from lvm-operator.

sp98 avatar sp98 commented on June 24, 2024

Hi @jgato We are planning to work on the documentation soon. In the meantime, you can use the following steps to create the persistent volumes.

  1. After deploying the operator, create LVMCluster resource
apiVersion: lvm.topolvm.io/v1alpha1
kind: LVMCluster
metadata:
name: lvmcluster-sample
namespace: lvm-operator-system    (This is the namespace you are running the operator on)
spec:
deviceClasses:
- name: vg1

  1. Wait for the topolvm-node daemonset pods to be running. ( it might restart a few times waiting for the volume group to be created)

  2. Deploy a sample application with PVC and storageclass.

apiVersion: storage.k8s.io/v1
metadata:
  name: topolvm-provisioner-vg1
provisioner: topolvm.cybozu.com
parameters:
  "csi.storage.k8s.io/fstype": "xfs"
  "topolvm.cybozu.com/device-class": "vg1"
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc1
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: topolvm-provisioner-vg1
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: topo-test
  labels:
    app: topo-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: topo-test
  template:
    metadata:
      labels:
        app: topo-test
    spec:
      containers:
        - name: topo-test
          image: ubuntu:18.04
          command: ["sleep"]
          args: ["3000000"]
          volumeMounts:
            - name: topo
              mountPath: /data/
      volumes:
        - name: topo
          persistentVolumeClaim:
            claimName: pvc1

from lvm-operator.

jgato avatar jgato commented on June 24, 2024

many thanks I will try this afternoon. Meanwhile, I am wondering how should I point to the physical device in the host?

from lvm-operator.

sp98 avatar sp98 commented on June 24, 2024

Right now you don't have to do anything to point to specific disks on the host. The operator takes all the disks that don't have filesystem and partitions.

For example, if you have /dev/sda and /dev/sdb in your host that don't have any filesystem or partitions on it, volume groups will use these disk.

from lvm-operator.

jgato avatar jgato commented on June 24, 2024

Ok I have created the LVMCluster but nothing seems to happens. The first strange thing I have noticed is, that it does not create any daemonset as you mentioned:

$> oc get daemonsets.apps -n lvm-operator-system 
No resources found in lvm-operator-system namespace.
$> oc describe lvmclusters.lvm.topolvm.io 
Name:         lvmcluster-sample
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  lvm.topolvm.io/v1alpha1
Kind:         LVMCluster
Metadata:
  Creation Timestamp:  2022-01-21T09:13:16Z
  Generation:          1
  Managed Fields:
    API Version:  lvm.topolvm.io/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:deviceClasses:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2022-01-21T09:13:16Z
  Resource Version:  171955
  UID:               298b48ba-561a-4422-9157-937175c208ca
Spec:
  Device Classes:
    Name:  vg1
Events:    <none>

Maybe because of the device class? In this host I have some available disks:

# lsblk 
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda       8:0    0   2.7T  0 disk 
sdb       8:16   0 931.5G  0 disk 
|-sdb1    8:17   0     1M  0 part 
|-sdb2    8:18   0   127M  0 part 
|-sdb3    8:19   0   384M  0 part /boot
`-sdb4    8:20   0   931G  0 part /sysroot
nvme0n1 259:0    0 745.2G  0 disk 
nvme1n1 259:1    0 745.2G  0 disk

from lvm-operator.

jgato avatar jgato commented on June 24, 2024

Not it is working and it has recognized two disks under the VG:

$> pvs
  PV           VG  Fmt  Attr PSize   PFree  
  /dev/nvme0n1 vg1 lvm2 a--  745.21g 745.21g
  /dev/sda     vg1 lvm2 a--   <2.73t  <2.73t

But the nvme1n1 is not included in the group. And it is also an available group

$> lsblk 
NAME    MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda       8:0    0   2.7T  0 disk 
sdb       8:16   0 931.5G  0 disk 
|-sdb1    8:17   0     1M  0 part 
|-sdb2    8:18   0   127M  0 part 
|-sdb3    8:19   0   384M  0 part /boot
`-sdb4    8:20   0   931G  0 part /sysroot
nvme0n1 259:0    0 745.2G  0 disk 
nvme1n1 259:1    0 745.2G  0 disk

Looking to the logs I can see the disk nvme1n1 under the same conditions of disk nvme0n1 which is included.

{"level":"info","ts":1642757843.7951472,"logger":"controller.lvmvolumegroup.vg-manager","msg":"listing block devices","reconciler group":"lvm.topolvm.io","reconciler kind":"LVMVolumeGroup","name":"vg1","namespace":"lvm-operator-system","VGName":"vg1"}
{"level":"info","ts":1642757843.8154957,"logger":"controller.lvmvolumegroup.vg-manager","msg":"does not match filter","reconciler group":"lvm.topolvm.io","reconciler kind":"LVMVolumeGroup","name":"vg1","namespace":"lvm-operator-system","Device.Name":"sda","filter.Name":"noFilesystemSignature"}
{"level":"info","ts":1642757843.8248024,"logger":"controller.lvmvolumegroup.vg-manager","msg":"does not match filter","reconciler group":"lvm.topolvm.io","reconciler kind":"LVMVolumeGroup","name":"vg1","namespace":"lvm-operator-system","Device.Name":"sdb","filter.Name":"noChildren"}
{"level":"info","ts":1642757843.8248734,"logger":"controller.lvmvolumegroup.vg-manager","msg":"does not match filter","reconciler group":"lvm.topolvm.io","reconciler kind":"LVMVolumeGroup","name":"vg1","namespace":"lvm-operator-system","Device.Name":"nvme0n1","filter.Name":"noFilesystemSignature"}
{"level":"info","ts":1642757843.8249245,"logger":"controller.lvmvolumegroup.vg-manager","msg":"does not match filter","reconciler group":"lvm.topolvm.io","reconciler kind":"LVMVolumeGroup","name":"vg1","namespace":"lvm-operator-system","Device.Name":"nvme1n1","filter.Name":"noFilesystemSignature"}
#

I used this disk for a test with a LocalStorage Operator. But now it seems that is free. Is there any way I can check why this disk is not added?

from lvm-operator.

sp98 avatar sp98 commented on June 24, 2024

what's the output of lsblk -o NAME,ROTA,TYPE,SIZE,MODEL,VENDOR,RO,RM,STATE,KNAME,SERIAL,PARTLABEL,FSTYPE for nvme1n1 disk?

from lvm-operator.

sp98 avatar sp98 commented on June 24, 2024

also, the output for cat /proc/1/mountinfo | grep nvme1n1

from lvm-operator.

jgato avatar jgato commented on June 24, 2024

yes, it still has a format:

#  lsblk -o NAME,ROTA,TYPE,SIZE,MODEL,VENDOR,RO,RM,STATE,KNAME,SERIAL,PARTLABEL,FSTYPE
NAME                                           ROTA TYPE   SIZE MODEL                                    VENDOR   RO RM STATE   KNAME   SERIAL             PARTLABEL  FSTYPE
sda                                               1 disk   2.7T LOGICAL VOLUME                           HP        0  0 running sda     PDNLL0ARH9D05M                LVM2_member
|-vg1-bd0d4467--e58d--4806--a453--90454e92a1da    1 lvm      5G                                                    0  0 running dm-0                                  
|-vg1-381eb9d7--a8dd--404d--9810--52ded982648b    1 lvm      1G                                                    0  0 running dm-1                                  
|-vg1-e04008f1--6eec--426a--aa8b--b9915c2911da    1 lvm      1G                                                    0  0 running dm-2                                  
|-vg1-358bcb04--7684--49d1--9bdf--945413b04cd7    1 lvm      1G                                                    0  0 running dm-3                                  
|-vg1-c93f3766--98fa--446a--a64f--13a6384c8f2a    1 lvm      1G                                                    0  0 running dm-4                                  
|-vg1-61628484--7f35--4d01--9dc2--86ce47c5fd55    1 lvm      1G                                                    0  0 running dm-5                                  
`-vg1-8ab73406--f756--4442--a567--e2d53958c2cd    1 lvm      1G                                                    0  0 running dm-6                                  
sdb                                               1 disk 931.5G LOGICAL VOLUME                           HP        0  0 running sdb     PDNLL0ARH9D05M                
|-sdb1                                            1 part     1M                                                    0  0         sdb1                       BIOS-BOOT  
|-sdb2                                            1 part   127M                                                    0  0         sdb2                       EFI-SYSTEM vfat
|-sdb3                                            1 part   384M                                                    0  0         sdb3                       boot       ext4
`-sdb4                                            1 part   931G                                                    0  0         sdb4                       root       xfs
nvme0n1                                           0 disk 745.2G MT0800KEXUU                                        0  0 live    nvme0n1 PHFT640100R2800CGN            LVM2_member
nvme1n1                                           0 disk 745.2G MT0800KEXUU                                        0  0 live    nvme1n1 PHFT640100PS800CGN            xfs

I thought that something in the log of the controller would have pointed to that.
Thanks for all, I continue doing some tests.

from lvm-operator.

sp98 avatar sp98 commented on June 24, 2024

I thought that something in the log of the controller would have pointed to that. Thanks for all, I continue doing some tests.

I think the vgmanager controller might have pointed in out in the first reconcile. The output of the controller that you have pasted above, might be from a later reconcile when both nvme0n1 and nvme1n1 have the filesystem.

from lvm-operator.

jgato avatar jgato commented on June 24, 2024

Once that I made nvme1n1 to not have a fsystem, is the operator detecting the new available disks to include it in the VolumeGroup?, or do I have to create a new LVMCluster to detect it?

from lvm-operator.

jgato avatar jgato commented on June 24, 2024

Something extrange happend while making a test. I wanted to try adding new disks to the existing LVMCluster or to a new one.
I added some more disks in the server, and after rebooting... these new disks have been added to the VG that I already had created previously (vg1)

(sdc and sde are the new disks that now appear under the same group)

[root@master-0 core]# vgs
  VG  #PV #LV #SN Attr   VSize VFree 
  vg1   5   4   0 wz--n- 6.00t <6.00t
[root@master-0 core]# pvs
  PV           VG  Fmt  Attr PSize   PFree  
  /dev/nvme0n1 vg1 lvm2 a--  745.21g 745.21g
  /dev/nvme1n1 vg1 lvm2 a--  745.21g 745.21g
  /dev/sda     vg1 lvm2 a--   <2.73t   2.72t
  /dev/sdc     vg1 lvm2 a--  931.48g 931.48g
  /dev/sde     vg1 lvm2 a--  931.48g 931.48g

At the same time I have created a new LVMCluster, because I expected this new groups would create a new VG (vg2) with the new disks. But it is not happening in that way.

Now there are two LVMClusters but none of them are using the new devices:

The LVMCluster that I created first (before adding new disks)

$ oc get  lvmcluster lvmcluster-sample -o yaml                                                                                                                                                                                  
apiVersion: lvm.topolvm.io/v1alpha1                                                                                                                                                                                                                           
kind: LVMCluster                   
metadata:       
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"lvm.topolvm.io/v1alpha1","kind":"LVMCluster","metadata":{"annotations":{},"name":"lvmcluster-sample","namespace":"lvm-operator-system"},"spec":{"deviceClasses":[{"name":"vg1"}]}}
  creationTimestamp: "2022-02-05T18:56:54Z"                                                                                                                                                                                                                   
  finalizers:                              
  - lvmcluster.topolvm.io
  generation: 1          
  name: lvmcluster-sample
  namespace: lvm-operator-system
  resourceVersion: "6276722"
  uid: fb91728e-1300-4fe1-9041-78d5a8e166d6
spec:
  deviceClasses:
  - name: vg1
status:
  deviceClassStatuses:
  - name: vg1
    nodeStatus:
    - devices:
      - /dev/nvme0n1
      - /dev/nvme1n1
      - /dev/sda
      node: master-0.apollo2.hpecloud.org
      status: Ready
  ready: true

And the new one:

[jgato@infra2 lvm-cr]$ oc get  lvmcluster lvmcluster-sample-2 -o yaml                                                           
apiVersion: lvm.topolvm.io/v1alpha1
kind: LVMCluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"lvm.topolvm.io/v1alpha1","kind":"LVMCluster","metadata":{"annotations":{},"name":"lvmcluster-sample-2","namespace":"lvm-operator-system"},"spec":{"deviceClasses":[{"name":"vg2"}]}}
  creationTimestamp: "2022-02-05T19:42:38Z"
  finalizers:
  - lvmcluster.topolvm.io
  generation: 1
  name: lvmcluster-sample-2
  namespace: lvm-operator-system
  resourceVersion: "6286193"
  uid: 61feced0-e258-4d65-ae4b-ec2c8bada8e1
spec:
  deviceClasses:
  - name: vg2
status:
  deviceClassStatuses:
  - name: vg1
    nodeStatus:
    - devices:
      - /dev/nvme0n1
      - /dev/nvme1n1
      - /dev/sda
      node: master-0.apollo2.hpecloud.org
      status: Ready

The outcome of the second one seems a little confusing. Showing information about the VG1.

How is the correct way of adding new disks?

from lvm-operator.

nbalacha avatar nbalacha commented on June 24, 2024

As of now, we only support a single LVMCluster which will use all disks on all nodes, which is why the newly added disks were added to the original volumegroup. Going forward, we will be adding features to all allow us to specify multiple volumegroups and the disks that need to be used for each.

Please open issues describing features and use cases which you would like to see in the operator. We welcome feedback.

from lvm-operator.

nbalacha avatar nbalacha commented on June 24, 2024

To add more info:
The proposed CR changes, once implemented, would be something as follows:

apiVersion: lvm.topolvm.io/v1alpha1
kind: LVMCluster
metadata:
name: lvmcluster-sample
namespace: lvm-operator-system (This is the namespace you are running the operator on)
spec:
deviceClasses:

  • name: vg1
    disks:
    • /dev/sdb
  • name: vg2
    disks:
    • /dev/sdc

from lvm-operator.

jgato avatar jgato commented on June 24, 2024

Ok, I understand that only one LVMCluser is allowed by the moment. But, this will LVMCluster got control on new disks appearing in the cluster? if you look to my previous post, the new added disks have been added to the VG (vg1), there are 5 disks. I guess, it is working at level of LVM or topolvm. But in the Openshift (SNO) Cluster, the LVMCluster wit the vg1 does not seems to be aware of the new disks, and it only lists 3 disks.

Let me know if you prefer to discuss this in a new thread.

from lvm-operator.

nbalacha avatar nbalacha commented on June 24, 2024

Yes, please open a new issue /thread for this.

from lvm-operator.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.