beornf / linstor-docker-volume Goto Github PK

Hi beornf,

I saw this repo because you sent a PR for the linstor-server. This is awesome! I'm the maintainer of the "official" linstor-docker-volume plugin, and I also co-maintain golinstor. So some comments/suggested improvements:

The "old" golinstor uses the linstor-client to do its actions. I saw that you use the "lade.io" repos to get the client. This version is very outdated and I guess it would not even work with a recent linstor controller. The old one used probuf + a custom protocol, recent client and server use a REST-API.

The good news is that golinstor now also provides a REST client. It is pretty "low level", but IMO the way forward (we already transitioned our CSI plugin to it). We might remove/or at least substantially change the high level abstraction of golinstor.

Do you want to work on the plugin together? Maybe we could eventually put it under some "LINBIT umbrella" and make this one the default. For historic reasons the official one started in Python, but I'd be all in to switch it to Go. Does any of that sound interesting? :-D.

I'd like to have this plugin - which should become the default for LINBIT - as much of a drop in replacement for the old python based plugin as possible. So I would like it to have the same CLI options as defined here: https://github.com/LINBIT/linstor-docker-volume/blob/master/README

Hey there,

researching how we can get different docker containers to share the same linstor volume.

Initially, running the first container to use the linstor test volume works properly and the container has proper access to volume.
If we try to run a different container we get the following error.

[root@do2 linstor-docker-volume]# docker run -it -v test:/mnt/shared/ nginx sh
docker: Error response from daemon: error while mounting volume '': VolumeDriver.Mount: unable to get exclusive open on /dev/drbd1003.
ERRO[0000] error waiting for container: context canceled

Seems that this has been added in CSI plugin which unfortunately we can't yet use in Docker Swarm.
piraeusdatastore/linstor-csi#35
piraeusdatastore/linstor-csi@b9372af
piraeusdatastore/linstor-csi@3890953

However, adding changes of b9372aff07e7664608f6080619bcf2f5cf3c8436 to driver.go leads to another error

[root@do2 ~]# docker run -it -v test:/mnt/shared/ nginx sh
docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/1d8eed5885c19913f0fcb283ac00efad1e79281d2c802f837a869fdc20c231b2/propagated-mount/test/data': VolumeDriver.Mount: mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t xfs /dev/drbd1003 /var/lib/docker-volumes/linstor/test
Output: mount: /var/lib/docker-volumes/linstor/test: /dev/drbd1003 already mounted on /var/lib/docker-volumes/linstor/test.

Any thoughts on how we could possibly tackle this would be much appreciated.

In my environment I configured a highly-available LINSTOR controller as per the instructions here: https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-linstor_ha

I had attempted to configure this plugin with multiple, comma-separated, LINSTOR Controllers like so:

[global]
storagepool = lvm-thin 
fs = xfs 
size = 100MB
replicas = 2
controllers = linstor://192.168.221.60,linstor://192.168.221.61,linstor://192.168.221.62

This would result in the below error unless the controller was running on the first listed address (192.168.221.60).

root@linstor-0:~# docker volume create --opt fs=xfs --opt size=1G -d linbit/linstor-docker-volume test03
Error response from daemon: create test03: VolumeDriver.Create: Post "http://192.168.221.60:3370/v1/resource-definitions": dial tcp 192.168.221.60:3370: connect: connection refused

I was able to work around the issue fairly easily, by configuring drbd-reactor to use a virtual IP address and just set that within the docker-volume plugin configuration. Still, it would be nice if this wasn't necessary and the docker-volume plugin could "round-robin" attempts to different controllers much like the linstor-client can be configured to do.

Consider this a feature request. Thanks a bunch for the work on this plugin!

So far diskless assignments get created "on demand" when needed. Our other plugins also delete diskless assignments when they are not needed anymore. It is not really a requirement for this plugin, but it is a good idea in general where we have like 10 hypervisor nodes where only one at a time accesses the data. There we really do not want to keep track of 9 unused diskless clients.

Test environment：

1.ARM64
2.root@node43:/home/horatio# cat /proc/version
Linux version 4.15.0-76-generic (buildd@bos02-arm64-060) (gcc version 7.4.0 (Ubuntu/Linaro 7.4.0-1ubuntu1~18.04.1)) #86-Ubuntu SMP Fri Jan 17 17:25:58 UTC 2020

Docker plugin install information:
root@node43:/home/horatio# docker plugin install lade/linstor
Plugin "lade/linstor" is requesting the following privileges:

• network: [host]
• mount: [/dev]
• mount: [/etc/linstor]
• allow-all-devices: [true]
• capabilities: [CAP_SYS_ADMIN]
  Do you grant the above permissions? [y/N] y
  latest: Pulling from lade/linstor
  Digest: sha256:a938f71a27be81b465490f8069b99cb95bc1887bcde106a00972b4cf65c474b1
  1e3a8ee5ca2d: Complete
  Error response from daemon: dial unix /run/docker/plugins/21a145a416b2f67311d44578dcf6496f91ec13dcae7c45fd62d4b9410f5b1ea1/linstor.sock: connect: no such file or directory

It looks like the LINSTOR socket problem.

Docker plugin list show:
root@node43:/home/horatio# docker plugin list
ID NAME DESCRIPTION ENABLED
21a145a416b2 lade/linstor:latest Linstor volume plugin for Docker false

I try to enable the plugin, it print the following information：
root@node43:/home/horatio# docker plugin enable 21a145a416b2
Error response from daemon: dial unix /run/docker/plugins/21a145a416b2f67311d44578dcf6496f91ec13dcae7c45fd62d4b9410f5b1ea1/linstor.sock: connect: no such file or directory

Configuration:
root@node43:/home/horatio# cat /etc/linstor/docker-volume.conf
[global]
controllers=192.168.1.10

Create volume test:
root@node43:/home/horatio# docker volume create -d lade/linstor:latest --opt fs=xfs --opt size=200M lsvol1
Error response from daemon: create lsvol1: error looking up volume plugin lade/linstor:latest: plugin lade/linstor:latest found but disabled

If anyone need more information,please let me know!
Thanks

So far the plugin handles all the LINSTOR resources that it finds. This can be reproduced by manually creating LINSTOR/DRBD resources and then calling the plugin's List().

For our other plugins we usually handle this by:

• setting a plugin-unique, so called "aux property", on the resource when it is created.
• on listing/removing/... we check if this property (i.e., a flag) is set and only do actions on resources that have this flag.

It would be nice to reuse the same aux prop we have in the Python plugin: https://github.com/LINBIT/linstor-docker-volume

I will add some other feature requests. Let's do it like this: If you like to work on a topic, somehow "claim it" so that I know you work on it, otherwise add some "help wanted" tag and I will take it.

As the title says, wanted to ask if it's possible to create a filesystem upon volume creation or mounting?

Now, when creating a volume and try to mount it to a container, I get an error described below,

Output: mount: /var/lib/docker-volumes/linstor/test: wrong fs type, bad option, bad superblock on /dev/drbd1001, missing codepage or helper program, or other error.

Once i run mkfs.xfs /dev/drbd1001 then mounting is getting done properly.