projectatomic / container-storage-setup Goto Github PK

View Code? Open in Web Editor NEW

153.0 26.0 82.0 482 KB

Service to set up storage for Docker and other container systems

License: Apache License 2.0

Shell 92.35% Makefile 1.07% Roff 6.58%

docker containers

container-storage-setup's People

Contributors

Stargazers

Watchers

container-storage-setup's Issues

not extending partition table

With docker-1.8.2-10.el7.centos.x86_64 at least in AWS (ami-3fb05d5f in us-west-2), I'm not getting the partition table being resized after boot. Will try to add some more debugging info here when I get a chance.

Provide a new option STORAGE_DRIVER

Right now docker-storage-setup assumes devicemapper driver. Provide a new
option STORAGE_DRIVER where one can specify other drivers like overlay.

There are few benefits of this.

- We want to avoid people to modify /etc/sysconfig/docker-storage directly
  as it can conflict with what docker-storage-setup script wants to do.
  By providing this option, those who want to use overlay, should be able
  to just edit /etc/sysconfig/docker-storage-setup and start the service.

- One should be able to override STORAGE_DRIVER="". This will indicate that
  don't setup any storage driver. This is for the people who want a way to
  disable docker-storage-setup.

Also add support for configuring overlay through this option.

And in the end fail if user is switching drivers and one is already configured.

Reserve 60% of free space by default for data volume

Right now by default we reserve 98% of free space as data volume. Now we
have enabled automatic pool extension logic. That will make sure that pool
data and metadata volumes will grow automatically as they start filling up.

This means we don't have to consume all the free space available at startup.
It can be consumed dynamically. One advantage of not consuming all the space
up front is that it can be used for some other purpose. For example, one
can try to grow root volume. 3G of root might not be sufficient.

Leaving free space also allows for extension of metadata volume later easily.
Some openshift folks complained that not enough metadata size is being
reserved at startup time. Leaving space free and growing volumes dynamically
gives us lot of flexibility here for different kind of workloads.

docker storage limited to 2TB by sfdisk

I tried to use docker-storage-setup on a disk that was larger than 2TB:

# lsblk
NAME           MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda              8:0    0 558.4G  0 disk 
├─sda1           8:1    0     1G  0 part /boot
└─sda2           8:2    0 557.4G  0 part 
  ├─rhel7-root 253:0    0 556.3G  0 lvm  /
  └─rhel7-swap 253:1    0     1G  0 lvm  [SWAP]
sdb              8:16   0   2.2T  0 disk

root@bkr-hv02: ~ # systemctl start docker-storage-setup
Job for docker-storage-setup.service failed because the control process exited with error code. See "systemctl status docker-storage-setup.service" and "journalctl -xe" for details.

It failed to start and threw an error that comes from the sfdisk utility used to create partitions.

root@bkr-hv02: ~ # systemctl status docker-storage-setup -l
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Wed 2016-04-06 11:36:58 EDT; 7s ago
  Process: 61143 ExecStart=/usr/bin/docker-storage-setup (code=exited, status=1/FAILURE)
 Main PID: 61143 (code=exited, status=1/FAILURE)

Apr 06 11:36:58 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61143]: /dev/sdb4             0         -          0   0  Empty
Apr 06 11:36:58 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61143]: Warning: partition 1 has size 2.4 TB (2398201315328 bytes),
Apr 06 11:36:58 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61143]: which is larger than the 2199023255040 bytes limit imposed
Apr 06 11:36:58 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61143]: by the DOS partition table for 512-byte sectors
Apr 06 11:36:58 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61143]: sfdisk: I don't like these partitions - nothing changed.
Apr 06 11:36:58 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61143]: (If you really want this, use the --force option.)
Apr 06 11:36:58 bkr-hv02.lab.eng.rdu.redhat.com systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=1/FAILURE
Apr 06 11:36:58 bkr-hv02.lab.eng.rdu.redhat.com systemd[1]: Failed to start Docker Storage Setup.
Apr 06 11:36:58 bkr-hv02.lab.eng.rdu.redhat.com systemd[1]: Unit docker-storage-setup.service entered failed state.
Apr 06 11:36:58 bkr-hv02.lab.eng.rdu.redhat.com systemd[1]: docker-storage-setup.service failed.

As suggested in the error, I added --force to /usr/bin/docker-storage-setup:

# diff -pruN docker-storage-setup /usr/bin/docker-storage-setup 
--- docker-storage-setup        2016-04-06 11:41:44.519253366 -0400
+++ /usr/bin/docker-storage-setup       2016-04-06 11:41:49.431230315 -0400
@@ -568,7 +568,7 @@ create_disk_partitions() {
     #   * Error handling when partition(s) already exist
     #   * Deal with loop/nbd device names. See growpart code
     size=$(( $( awk "\$4 ~ /"$( basename $dev )"/ { print \$3 }" /proc/partitions ) * 2 - 2048 ))
-    cat <<EOF | sfdisk $dev
+    cat <<EOF | sfdisk --force $dev
 unit: sectors

 ${dev}1 : start=     2048, size=  ${size}, Id=8e

And then it was able to create the partition:

# systemctl status docker-storage-setup -l
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: inactive (dead)

Apr 06 11:42:07 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61555]: Volume group "docker_vg" successfully created
Apr 06 11:42:07 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61555]: Rounding up size to full physical extent 192.00 MiB
Apr 06 11:42:07 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61555]: Wiping xfs signature on /dev/docker_vg/docker-poolmeta.
Apr 06 11:42:07 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61555]: Logical volume "docker-poolmeta" created.
Apr 06 11:42:08 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61555]: Logical volume "docker-pool" created.
Apr 06 11:42:08 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61555]: WARNING: Converting logical volume docker_vg/docker-pool and docker_vg/docker-poolmeta to pool's data and metadata volumes.
Apr 06 11:42:08 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61555]: THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Apr 06 11:42:08 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61555]: Converted docker_vg/docker-pool to thin pool.
Apr 06 11:42:08 bkr-hv02.lab.eng.rdu.redhat.com docker-storage-setup[61555]: Logical volume "docker-pool" changed.
Apr 06 11:42:08 bkr-hv02.lab.eng.rdu.redhat.com systemd[1]: Started Docker Storage Setup.

# lsblk
NAME                             MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                8:0    0 558.4G  0 disk 
├─sda1                             8:1    0     1G  0 part /boot
└─sda2                             8:2    0 557.4G  0 part 
  ├─rhel7-root                   253:0    0 556.3G  0 lvm  /
  └─rhel7-swap                   253:1    0     1G  0 lvm  [SWAP]
sdb                                8:16   0   2.2T  0 disk 
└─sdb1                             8:17   0 185.5G  0 part 
  ├─docker_vg-docker--pool_tmeta 253:2    0   192M  0 lvm  
  │ └─docker_vg-docker--pool     253:4    0  74.1G  0 lvm  
  └─docker_vg-docker--pool_tdata 253:3    0  74.1G  0 lvm  
    └─docker_vg-docker--pool     253:4    0  74.1G  0 lvm

Add capability to setup lvm thin pool and pass it to docker

Right now we setup a data lvm and metadata lvm and pass it to docker and docker in turn sets up thin pool using libdevmapper library. Now lvm tools support creation of thin pool and one can create a
thin pool using lvm and pass it to docker.

Advantage of this approach is that lvm can do all the maintenace of pool like automatic resizing and
doing metadata checks when pool is activated etc.

So this should turn out to be a better approach.

Add --dry-run option to docker-storage-setup

Most LVM commands have a dry-run option, would it be possible to add such an option in docker-storage-setup so that it would tell me what it would have done ?

add a small help message

Recently Chris Evich got surpised when he typed "docker-storage-setup -h" and instead of displaying any help message it went ahead and setup the storage.

By default reserve lesser space for data volume

Right now DATA_SIZE is 60%FREE. That is use 60% of free space for data volume. Dan Walsh was saying that this sounds like a high percentage. It might be a good idea to drop it down a bit. Say 40%FREE.

Provide capability to specify Minimum data volume size

Currently our default is to create data volume of size 60%FREE. That is use 60% free space in volume group to create data volume.

But on some systems, there is very small free space available in root volume group (say 128MB) and
in that case we end up creating a very small thin pool by default. (say 80MB).

This results in either docker start failure or docker hangs when first image is pulled. So too small a pool out of box is not good.

Provide a knob to specify minimum data volume size and keep the default as 2G. So if minimum required free space is not available, then pool creation will fail and docker will fall back to using loop devices.

Should docker start or fail if docker-storage-setup fails

Currently if docker-storage-setup fails, docker still starts and falls back to using loopback devices. Is this the right behavior. Or docker should fail to as docker-storage-setup failed.

Opening this issue for having a conversation on this topic.

Root filesystem is not grown

If I'm reading the script correctly only the root logical volume is resized not the filesystem on top of it. Is the user expected to grow that after the fact or as needed?

cleanup related to growpart

docker-storage-setup script does multiple things. Not only it sets up an lvm thin pool, but it can create partitions on disk and also automatically grow root pvs upon reboot. This is highly custom for project atomic.

It is desirable to be able to use this service outside project atomic too in a more generic settings where auto growing is not required and growpart might not even be available.

Write a patch series which does few things.

Do not make a dependency on growpart. Use it if available.
Disable growpart logic by default and provide a knob to enable it through config file.

Use option -y for lvcreate

docker-storage-setup: Use option -y for lvcreate

lvcreate creates data and metadata volumes before they are converted in
thin pool volume. If lvcreate finds an existing signature, it asks the
question whether to wipe it or not. As we run as a service by default
answer is no. But that signature gets wiped out anyway. Reason being
that lvcreate zeroes first 4K of logical volume and as part of that
zeroing any signatures in first 4K will get wiped out.

IOW, as of now, it does not matter what's the answer. We are wiping out
signatures by default during lvcreate.

And there are following messages on console.

docker-storage-setup[3240]: WARNING: xfs signature detected on /dev/rhel/docker-pool at offset 0. Wipe it? [y/n]: n
docker-storage-setup[3240]: Aborted wiping of xfs.
docker-storage-setup[3240]: 1 existing signature left on the device.
docker-storage-setup[3240]: Logical volume "docker-pool" created.
We really don't seem to have a need to preserve signatures. At pvcreate,
time we can make sure that disk already does not have any data on it. Once
it has been added to volume group, I guess we don't have to worry about
signatures existing and we can wipe it by default.

So let us use "-y" for lvcreate. This should kill those messages which
ask the question and wipe signatures anyway and give explicit messages that
signatures are being wiped. Atleast there will be less confusion and we know
that by default we will wipe signature during lvcreate.

Enforce devicemapper graphdriver in generated /etc/sysconfig/docker-storage file

Right now docker-storage-setup creates storage options which are valid
only for devicemapper driver. If some other driver is trying to initialize
and it is not properly written it will return error saying unrecognized
options dm.fs etc.

I am running into this issue now. Recently upstream has added zfs as
graphdriver and that takes precedence over devicemapper. And that driver
is returning error that dm.fs is unrecognized option.

Given that docker-storage-setup is expecting docker to use devicemapper
graphdriver and generating options which only work with devicemapper,
it makes sense to enforce the driver to use here to avoid errors.

Add option to add more options to docker storage

If say I want to pass e.g. --storage-opt dm.basesize=50G to docker, currently the easiest way to do that is to change /etc/sysconfig/docker, where one would expect at least to also be able to change (also) /etc/sysconfig/docker-storage-setup where other options are specified.

Alternatively, we could figure out how to sanely merge human and automatic changes in /etc/sysconfig/docker-storage, but that's nontrivial, and I prefer not mixing human-edited and machine-edited files.

Delete non-pool code paths

Since moby/moby#12404 we should just drop the non-pool code paths for distribution targets which don't need them, as users should not use it.

docker-storage-setup fails but has already gone half way

docker-1.7.1-108.el7.x86_64

Trying to swap between overlay and devicemapper backend...

Added a new disk to the VM, and edited /etc/sysconfig/docker-storage-setup so it contains:
DEVS=/dev/vdc
VG=docker

Ran docker-storage-setup and...

# docker-storage-setup 
  WARNING: Device for PV VP6BxI-8NYd-kpBe-Xlr7-LJLO-dmGo-zR2bSx not found or rejected by a filter.
Checking that no-one is using this disk right now ...
OK

Disk /dev/vdc: 416101 cylinders, 16 heads, 63 sectors/track
sfdisk:  /dev/vdc: unrecognized partition table type

Old situation:
sfdisk: No partitions found

New situation:
Units: sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/vdc1          2048 419430399  419428352  8e  Linux LVM
/dev/vdc2             0         -          0   0  Empty
/dev/vdc3             0         -          0   0  Empty
/dev/vdc4             0         -          0   0  Empty
Warning: partition 1 does not start at a cylinder boundary
Warning: partition 1 does not end at a cylinder boundary
Warning: no primary partition is marked bootable (active)
This does not matter for LILO, but the DOS MBR will not boot this disk.
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)
  Replacing dev /dev/vdc1 pvid VP6BxI8NYdkpBeXlr7LJLOdmGozR2bSx with dev /dev/vdc1 pvid GsLnMJ348U0S4TTr1l24QgrJYo2ZC1A4
  Physical volume "/dev/vdc1" successfully created
  Volume group "docker" successfully created
Storage is already configured with overlay driver. Can't configure it with devicemapper driver. To override, remove /etc/sysconfig/docker-storage and retry.

If it's going to error-out with some warning about existing overlay, why bother doing anything to the hard drives? I think you should move that check up to the front of the script.

if metadata lv already exists do not try to create a new one

I metadata lv already exists (from previous run), then do not try to create a new one as that will fail.

error out if /usr/bin/growpart is not available and GROWPART=true

If user wants to grow partitions and specified GROWPART=true and /usr/bin/growpart executable is not installed, then error out instead of continuing.

Wrong assumption about $ROOT_DEV is LVM

# pvcreate /dev/vda2 
# vgcreate docker /dev/vda2
# reboot
# echo 'VG=docker' > /etc/sysconfig/docker-storage-setup
# systemctl stop docker
# rm -rf /var/lib/docker
# docker-storage-setup
  Volume group "vda1" not found
  Cannot process volume group vda1

The problem is in

ROOT_DEV=$( awk '$2 ~ /^\/$/ && $1 !~ /rootfs/ { print $1 }' /proc/mounts )
ROOT_VG=$( lvs --noheadings -o vg_name $ROOT_DEV | sed -e 's/^ *//' -e 's/ *$//')

as ROOT_DEV becomes /dev/vda1 which is not LVM, and hence the script breaks.

Why is there an assumption that ROOT_DEV is LVM?

Devices /dev/cciss/c0d1 detected as partition

Hi,

Specifying DEVS="/dev/cciss/c0d1" which is a correct device label fails with the following error:
"Partition specification unsupported at this time."

The following on line 426

    if [[ $dev =~ .*[0-9]$ ]]; then
       Fatal "Partition specification unsupported at this time."
    fi

Identifies any number as a partition which is false in the above case.

/dev/cciss/c0d1 is the device name of a logical drive on an HP DL 380 server using a RAID controller.

My suggestion would be to remove the test for the number as it is not sufficient, and creates false answers.

Setting up overlay fails if there is no volume group specified

If user is trying to setup overlay driver then there is no need to enforce volume group check. Decouple these two.

consider a manual via mkdocs

I recently converted ostree (and rpm-ostree) to have mkdocs manuals. I think it's clear we should have some sort of larger documentation where we can cover storage design and the high level in a fashion that's not easy to do in a man page.

We can have a whole section on e.g. the background behind overlayfs.

This would duplicate the man page a little bit - but the man page I think should discuss the details of config options, whereas the manual would be more a "what is this anyways" thing.

ship a default /etc/sysconfig/doceker-storage-setup file

I am thinking that it might be a good idea to ship a default /etc/sysconfig/docker-storage-setup file which documents supported options. That way it becomes easier for user to find and modify options, especially if man pages are not around.

Add skeleton /etc/sysconfig/docker-storage-setup

We want to encourage the use of LVM to manage docker's storage. Our documentation instructs users to copy over /usr/lib/docker-storage-setup/docker-storage-setup to /etc/sysconfig/docker-storage-setup in order to customize defaults.

RHEL users are accustomed to looking in /etc/sysconfig to customize services, docker is no exception:

# rpm -qc docker
/etc/sysconfig/docker
/etc/sysconfig/docker-network
/etc/sysconfig/docker-storage

The lack of any /etc/sysconfig/docker-storage-setup in our package leads to some confusion about why that file does not exist.

This PR is to create a placeholder file at /etc/sysconfig/docker-storage-setup that instructs users where to find documentation on how to use docker-storage-setup, and brings docker-storage-setup in-line with the rest of the docker package's config files:

Add a facility to be able to prepare a separate volume for docker root

In case of overlay driver, all images and containers will be stored on rootfs. Some might not want that. Hence create options so that docker-storage-setup can prepare a separate volume and mount it somewhere and force docker to use that as root directory. That way all image/containers/volumes will be stored on a separate volume instead of rootfs.

Root fs should still grow despite some early errors

Couple of errors like, old mode detection and DEVS not specified while VG does not exist, should not prevent us from growing root fs. These errors are relevant only when we try to setup lvm thin pool. Delay these error checks.

Also do some minor code reorganization and cleanup.

slightly change of behavior after wiped off previous docker thinp and re-run

After wiped off previous docker thinp and start docker-storage-setup, it failed with,

INFO: Timed out waiting for device /dev/mapper/dss--testing--vg-docker--pool
Apr 05 10:27:27 aos-ci-dsstst-rhel-31xfap docker-storage-setup[3840]: ERROR: Already configured thin pool /dev/mapper/dss--testing--vg-docker--pool is not available. If thin pool is taking longer to activate, set DEVICE_WAIT_TIMEOUT to a higher value and retry

The message pretty much lead users to try DEVICE_WAIT_TIMEOUT, but instead they most likely need to delete /etc/sysconfig/docker-setup and re-run.

do not overwrite /etc/sysconfig/docker-storage always

Right now upon each run of script, we overwrite /etc/sysconfig/docker-storage
and this could be bad if user added some of his own storage options. These
will be lost over next run of script.

Write /etc/sysconfig/docker-storage only when pool is created. After that
leave it untouched.

enable specifying ROOT_DEV

I had hard times with this line:

ROOT_DEV=$( awk '$2 ~ /^\/$/ && $1 !~ /rootfs/ { print $1 }' /proc/mounts )

Had to manually change it to

ROOT_DEV="/dev/vdb"

in order to make the script work

Provide a way to avoid running out of metadata space on devicemapper thinp volume

There are been a handful of cases where users are running out of space on their metadata volumes.

This was mostly caused by lack of garbage collection in OpenShift and Kubernetes (which has since been implemented and resolved, lessening the urgency of this issue).

But, we want to do a better job avoiding this situation regardless, so ... based on feedback from internal and the field, we would like a way to automatically extend the metadata device using some free space remaining in the volume group.

See #28

/cc @rhvgoyal

Provide an option to specify user provided docker options

Right now storage options for docker are read from /etc/sysconfig/docker-storage. And this file is created by docker-storage-setup. There is no way a user can provide its own storage options to docker during first start itself.

For example, if one wants to specify dm.basesize or dm.fs or dm.mountopt, one can do it only after docker-storage-setup has run once. Some of these options like dm.basesize need to be specified during first run itself.

So provide a new option for docker-storage-setup, say USER_DOCKER_OPTIONS which will take an option string from user and simply append it to finally generated DOCKER_STORAGE_OPTIONS

Error when backing device does not exist

# rpm -q docker
docker-1.7.1-109.el7.x86_64

# grep ^DEVS /etc/sysconfig/docker-storage-setup
DEVS=/dev/vdx

# ./docker-storage-setup.sh 
./docker-storage-setup.sh: line 381: * 2 - 2048 : syntax error: operand expected (error token is "* 2 - 2048 ")
  Volume group "docker" not found
  Cannot process volume group docker
  Volume group "docker" not found
  Cannot process volume group docker

Enable deferred device removal feature in docker devicemapper graph driver

Docker 1.7 devicemapper graph driver now supports deferred device removal feature. That is if device backing image/container is busy, then docker will not wait for all device refcounts to be dropped. Instead docker will schedule a deferred a deferred removal on device. That is when last reference to
device is dropped, kernel will automatically remove the device.

Enable this feature by default if docker version supports it.

Create automated test cases

Create some simple automated test cases. Especially for syntax checking of different possible input types.

Need to create systemd unit to tell docker to wait for storage

Normally, LVM devices are probed pretty early, but it can happen that storage devices are slow to start. I have a box with ~ 20 TB across 20 disks, and sometimes on boot not all of them have spun up by the time docker starts.

We'll need to tweak d-s-s to generate a systemd drop-in in /etc/systemd/system/docker.service.d with Before=dev-mapper-docker--meta.device dev-mapper-docker--data.device

Fix location of docker-storage-setup.conf file

Currently docker-storage-setup.conf file is being installed in /usr/lib64/... instead of /usr/lib/. Fix it.

Also, script exits if growpart is not installed. I had expected it will continue. But because "set -e" is there, script things this is an error and exits. Fix that too.

Do not create partition on devices

I am wondering what's the need of creating partitions on devices passed in using DEVS=. Why not simply do pvcreate on these devices and add to volume group? This seems to unnnecessary increase the complexity of code and we have to use sfdisk. I think we should be able to drop it.

Allow data size suffix of type GB and GiB

With MIN_DATA_SIZE patch, we restricted suffixes to be of single character. But users reported that they also use syntax of type GB. Allow GB and GiB.

skip block zeroing in lvm thin pool

By default docker sets up a thin pool with block zeroing disabled in thin pool for performance reasons. As this script sets up a thin pool for docker, mimic docker behavior.

Device mapper developers also have recommended making use of skip block zeroing for performance reasons.

Provide capability to enable automatic docker pool extension

lvm provides capabilities to monitor the thin pool and extend it automatically. Use that capability and allow automatic extension of pool setup by docker-storage-setup by default.

Error out early if root device is not a logical volume

Right now d-s-s assumes that system is using lvm and root devices is a logical volume backed by a logical volume group. But that's not the case always. Detect this early and exit gracefully.

Adjust default chunksize on thinp volume created by docker-storage-setup

Although we expect most persistent I/O to be written to volumes attached to a container, we want the default docker graph storage on RHEL systems to be as performant as possible.

smallfile, random I/O performance improves when increasing thinp chunksize:

Increased chunksize also decreases metadata space usage for smallfile, random I/O:

The same is not the case for sequential read/write workloads:

The trade-off for reduced metadata space is increased space on the data thinp device:

The reason for this increased data usage under random I/O workloads is documented here:
http://people.redhat.com/msnitzer/2014-01-21-random-io-and-copy-on-write-schemes.markdown.html

disable auto data volume extension with DATA_SIZE is specified

We are still doing auto data volume extension when user specifies DATA_SIZE. Say DATA_SIZE=+10%FREE, then every reboot we will extend data volume by 10% of free space.

Given that we are moving towards relying on lvm to extend data volume for pool based on threshold, not sure how many people need this functionality.

There are two ways to handle it.

Get rid of auto extension for pool data volume completely. Rely on lvm. This might be little surprise over upgrade for an existing DATA_SIZE user. Having said that we don't publicize in man page that DATA_SIZE can take arguments which suit to "lvmextend -l". We have maintained that pass in arguments suitable to "lvextend -L". So chances of somebody relying on auto data extension on every reboot are slim.
Retain auto growth capability but make it conditional. Say AUTO_DATA_LV_GROW=1. By default turn it off in /etc/sysconfig/docker-storage-setup. For users upgrading d-s-s, they will have turned it on by default until they pass in AUTO_DATA_LV_GROW=0.

Right now I am planning to send patches for first approach. If that's not acceptable, I can work on second approach.

sfdisk: unrecognized option --show-pt-geometry

I see this on Fedora 22 at least.

docker-storage-setup selects swap logical volume for storage

reproduces on several vms running on top of rehv;

# systemctl status docker-storage-setup
● docker-storage-setup.service - Docker Storage Setup
   Loaded: loaded (/usr/lib/systemd/system/docker-storage-setup.service; disabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Sun 2016-01-03 22:57:08 IST; 1h 2min ago
  Process: 31474 ExecStart=/usr/bin/docker-storage-setup (code=exited, status=5)
 Main PID: 31474 (code=exited, status=5)

Jan 03 22:57:08 mtayer-centos7-5.eng.lab.tlv.redhat.com systemd[1]: Starting Docker Storage Setup...
Jan 03 22:57:08 mtayer-centos7-5.eng.lab.tlv.redhat.com docker-storage-setup[31474]: Rounding up size to full physical extent 44.00 MiB
Jan 03 22:57:08 mtayer-centos7-5.eng.lab.tlv.redhat.com docker-storage-setup[31474]: Volume group "vg0" has insufficient free space (0 extents): 11 required.
Jan 03 22:57:08 mtayer-centos7-5.eng.lab.tlv.redhat.com systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=5/NOTINSTALLED
Jan 03 22:57:08 mtayer-centos7-5.eng.lab.tlv.redhat.com systemd[1]: Failed to start Docker Storage Setup.
Jan 03 22:57:08 mtayer-centos7-5.eng.lab.tlv.redhat.com systemd[1]: Unit docker-storage-setup.service entered failed state.
Jan 03 22:57:08 mtayer-centos7-5.eng.lab.tlv.redhat.com systemd[1]: docker-storage-setup.service failed.

[root@mtayer-centos7-5 ~]# lvdisplay
  --- Logical volume ---
  LV Path                /dev/vg0/lv_swap
  LV Name                lv_swap
  VG Name                vg0
  LV UUID                zjSk0c-uMhb-x33C-YjFH-8tgO-4idb-dMFEqB
  LV Write Access        read/write
  LV Creation host, time mtayer-centos7-5.eng.lab.tlv.redhat.com, 2015-12-30 16:53:19 +0200
  LV Status              available
  # open                 2
  LV Size                4.00 GiB
  Current LE             1024
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:1

  --- Logical volume ---
  LV Path                /dev/vg0/lv_root
  LV Name                lv_root
  VG Name                vg0
  LV UUID                Qzt2Yr-VaWH-Phkp-p0GA-jemU-Co6K-IiblMx
  LV Write Access        read/write
  LV Creation host, time mtayer-centos7-5.eng.lab.tlv.redhat.com, 2015-12-30 16:53:19 +0200
  LV Status              available
  # open                 1
  LV Size                35.51 GiB
  Current LE             9090
  Segments               2
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     8192
  Block device           253:0

[root@mtayer-centos7-5 ~]# lsblk
NAME                                                                                     MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sr0                                                                                       11:0    1 1024M  0 rom  
vda                                                                                      252:0    0   40G  0 disk 
├─vda1                                                                                   252:1    0  500M  0 part /boot
└─vda2                                                                                   252:2    0 39.5G  0 part 
  ├─vg0-lv_root                                                                          253:0    0 35.5G  0 lvm  /
  └─vg0-lv_swap                                                                          253:1    0    4G  0 lvm  [SWAP]
loop0                                                                                      7:0    0  100G  0 loop 
└─docker-253:0-131825-pool                                                               253:2    0  100G  0 dm   
  └─docker-253:0-131825-7f214d2227a61922ad5afc282a4d59f1a5067f79b4a86a6f06d17a5938b87df3 253:3    0  100G  0 dm   
loop1                                                                                      7:1    0    2G  0 loop 
└─docker-253:0-131825-pool                                                               253:2    0  100G  0 dm   
  └─docker-253:0-131825-7f214d2227a61922ad5afc282a4d59f1a5067f79b4a86a6f06d17a5938b87df3 253:3    0  100G  0 dm   

# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core)

Cloud-Init triggered install/start of docker will always hang

If Cloud-init is used to trigger installs of docker, docker startup, and provisioning in general will just hang. This is irrespective of what the actual method of install/config is - ie. Puppet/Ansible etc.

Expanding on that:

In our method, we use cloud-init to trigger Puppet or Ansible for the initial configuration run.
This succeeds until the docker daemon is requested to start.
The docker daemon service requires docker-storage-setup to have already run.
The docker-storage-setup systemd config requires cloud-final.service ( ie. cloud-init) to have finished - https://github.com/projectatomic/docker-storage-setup/blob/master/docker-storage-setup.service#L3
The daemon start process just hangs at this point, because of this circular dependency.

The result is that provisioning cannot complete until cloud-init has failed/died - or the box has been rebooted.

Commenting out After=cloud-final.service makes this problem go away.

I understand the thinking behind the requirement - basically, ensure the box has had it's storage configured by Cloud-init, to be used by docker-storage-setup - but the existing restriction makes more sophisticated hands-off provisioning awkward.

I'm not sure what to suggest as a solution here, sorry!

Do not auto extend data lv. Let user use lvm facilities for this as per the need

Right now docker-storage-setup keeps on extending the size of data volume
as long as there is free space in the pool. So even if a user has left
some space in the pool free for metadata volume extension, that free space
will be consumed by data volume extension.

Now lvm2 has nice capabilities to automatically extend pool (both data and
metadata) when pool is running short on space. Following are two knobs which
one can use for auto extension.

thin_pool_autoextend_threshold = 100
thin_pool_autoextend_percent = 20

By default extension is disabled. One needs to enable it manually by chaning
the value of thin_pool_autoextend_threshold.

In general it is a good idea to know consume all the free pool space until
and use mechanism like extending pool by certain percentage when it hits
threshold. That allows free space in volume group to be shared by other
entities like root which might have to be grown on need basis too.

Allow partition devices in DEVS=

Right now docker-storage-setup does not allow partition devices in DEVS=. For people who don't use lvm on root disk and don't create a volume group, this can be problematic. Now they don't have an extra disk and they can create a paritition on root disk and pass to dss.

If we allow partition devices in DEVS=, then one can easily create a partition on root disk and use that to carve out thin pool.

docker-storage-setup fails to start: Volume group "vg0" has insufficient free space

Reproduces on four several machines all on top of RHEV;

# journalctl --unit docker-storage-setup
-- Logs begin at Sat 2016-01-02 12:22:39 IST, end at Sun 2016-01-03 11:45:42 IST. --
Jan 03 11:40:32 mtayer-centos7-1.eng.lab.tlv.redhat.com systemd[1]: Starting Docker Storage Setup...
Jan 03 11:40:33 mtayer-centos7-1.eng.lab.tlv.redhat.com docker-storage-setup[105963]: Rounding up size to full physical extent 20.00 MiB
Jan 03 11:40:33 mtayer-centos7-1.eng.lab.tlv.redhat.com docker-storage-setup[105963]: Volume group "vg0" has insufficient free space (0 extents): 5 required.
Jan 03 11:40:33 mtayer-centos7-1.eng.lab.tlv.redhat.com systemd[1]: docker-storage-setup.service: main process exited, code=exited, status=5/NOTINSTALLED
Jan 03 11:40:33 mtayer-centos7-1.eng.lab.tlv.redhat.com systemd[1]: Failed to start Docker Storage Setup.
Jan 03 11:40:33 mtayer-centos7-1.eng.lab.tlv.redhat.com systemd[1]: Unit docker-storage-setup.service entered failed state.
Jan 03 11:40:33 mtayer-centos7-1.eng.lab.tlv.redhat.com systemd[1]: docker-storage-setup.service failed.

# cat /etc/sysconfig/docker-storage-setup
# Edit this file to override any configuration options specified in
# /usr/lib/docker-storage-setup/docker-storage-setup.
#
# For more details refer to "man docker-storage-setup"


# vgdisplay 
  --- Volume group ---
  VG Name               vg0
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  4
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                3
  Open LV               3
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               19.51 GiB
  PE Size               4.00 MiB
  Total PE              4994
  Alloc PE / Size       4994 / 19.51 GiB
  Free  PE / Size       0 / 0   
  VG UUID               bwF5Mc-rh92-yRoz-FISB-PCQc-2feu-o419kX

# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/vg0-lv_root  9.1G  4.2G  4.5G  49% /
devtmpfs                 1.9G     0  1.9G   0% /dev
tmpfs                    1.9G     0  1.9G   0% /dev/shm
tmpfs                    1.9G  183M  1.7G  10% /run
tmpfs                    1.9G     0  1.9G   0% /sys/fs/cgroup
/dev/vda1                477M  141M  308M  32% /boot
/dev/mapper/vg0-lv_home  898M  2.3M  833M   1% /home
tmpfs                    379M     0  379M   0% /run/user/0

# rpm -qa | grep -E "docker|lvm"
docker-selinux-1.8.2-10.el7.centos.x86_64
lvm2-2.02.130-5.el7.x86_64
docker-1.8.2-10.el7.centos.x86_64
lvm2-libs-2.02.130-5.el7.x86_64

# cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core)

projectatomic / container-storage-setup Goto Github PK

container-storage-setup's People

Contributors

Stargazers

Watchers

Forkers

container-storage-setup's Issues

Recommend Projects

Recommend Topics

Recommend Org