As it stands today, around 74MB is downloaded from the Docker hub over the network while installing the plugin.
This is saved as a temporary file in LCFS.  After it is extracted to the plugin directory, it is running inside a Centos7.2.1511 container.
That container itself creates around 8000 files (inodes) in LCFS and around 300MB of data in LCFS.

Explore using Debian golang as a base image.

All these being done, just to run a binary of size 10MB.  Just wondering if this can be made smaller if that is converted to C.

PS:  The overhead is a lot smaller with V1 plugin model.

Hi, I created a Ubuntu 18.04 VM on Virtualbox, attached a virtual hard disk to it and then tried to install lcfs on that disk. Here is the error I am getting:

root@pensu-VirtualBox:~# curl -fsSL http://lcfs.portworx.com/lcfs-setup.sh | sudo DEV=/dev/sdb bash
Unable to find image 'centos:latest' locally
latest: Pulling from library/centos
aeb7866da422: Pull complete 
Digest: sha256:67dad89757a55bfdfabec8abd0e22f8c7c12a1856514726470228063ed86593b
Status: Downloaded newer image for centos:latest
Preparing...                          ########################################
Updating / installing...
lcfs-0.5.0-0                          ########################################
Connecting to device, please wait...
Note: LCFS device file exists. Using existing device file /dev/sdb without modifying.
lcfs[2187]: Build: 3e2810a7e7150a231e03d9e75c565c330484cc13 Release: 0.5.0-3e2810a
lcfs[2187]: Maximum memory allowed for data pages 512 MB
lcfs[2187]: Formatting /dev/sdb, size 16106127360
lcfs[2187]: /dev/sdb mounted at /var/lib/docker
lcfs[2187]: /dev/sdb mounted at /lcfs
lcfs: linux.h:43: lc_lockOwned: Assertion `(lock == ((void *)0)) || lock->__data.__writer || (!exclusive && lock->__data.__nr_readers)' failed.

And it hangs there forever, any idea why it could be happening? TIA. @jjobi

It would be good to provide a script or something to save existing images and containers and loading those to LCFS.

LCFS and plugin can be installed/built independently, so they should make sure compatibility before starting Docker services.

lcfs stats can request displaying stats for a layer or for all the layers in LCFS. But the stats printed to the logs of LCFS daemon - displayed on stdout in foreground mode and syslog in daemon mode. Modify this command to collect those from LCFS and display on stdout.

It would be good to provide a mapping from layer identifiers (the directories seen under /lcfs/lcfs) to actual image and container names/identifiers.

I am using --volumes-from, for example:

$ docker run -d -P --name web -v /webapp training/webapp python app.py

and then

$ docker run -d --volumes-from web -it --rm --name bb busybox sh

I do not see the first container's data in the second.

When LCFS is unmounted, it writes out free space extent map to disk. Make sure space for that is available, by reserving some space ahead. Applications will start getting ENOSPC when file system is almost 90% full, but need to make sure the file system can shut down properly as well.

It would be nice to have a fsck like utility to check the consistency of LCFS.

As of now, almost every task in LCFS is performed FUSE threads, with the exception of two global threads purging page caches and flushing dirty pages. If there is a worker queue support, we could offload many of the tasks which can be run concurrently and/or in the background without holding up fuse threads.

Currently, NaiveDiffDriver is used for finding changes made in a layer compared to its parent. Implement a better scheme similar to those used by union file systems.

As of now, free space extents are kept in a single list and that could become inefficient trying to find free space in file system. It would be better if those are grouped into buckets and space allocation operations can quickly find free extents with enough free space to satisfy the allocation.

Layers in an image and all containers running on top of those, share a common block page cache. Size that based on the amount of data stored in all of those.

Docker creates local volumes in root layer and LCFS is not optimized for those. Current assumption is that everything in root layer is Docker metadata for managing image and container layers. But volumes may add large amounts of data to it.

Ciao!

How are you?

I have build ofteh 6 containers - 15 hours at once with overlay2.
Each built container is about 50 gigabytes. That's is ok.
But when I commit all at once (in parallel), in 20 hoursm it hasn't commited any of one. Looks like either blocked or just slow.

Is lcfs faster with the docker commit? Beacuse it is very bad (of course not ssd, hdd of course, becasuse they are huge - for LEDE and OpenWrt).

Thanks,
Patrik

LCFS by default uses a maximum 512MB for block cache. That may be too small for certain workflows, so provide a command to change that limit.

LCFS uses locks for serializing between concurrent operations. Investigate if RCU would perform better than locks.

The following sequence of operations happen while creating/removing an image layer.

• Creating an image layer:

1. Create a layer, populate with data.
2. Update bunch of files in /var/lib/docker about the new image (the last one is repositories.json).

• Removing an image layer:

1. Update a bunch of files in /var/lib/docker (which also replaces repositories.json with new data).
2. Remove the layer corresponding to the image being deleted.

If the system crashes after step 1 during the above operations, there will be an orphaned image in the graphdriver. Need a utilitly to clean up such orphaned layers.

See #14 on how Docker keeps track of layers.

I am having the following problem:

kir@kd:~$ go get github.com/portworx/lcfs
package github.com/portworx/lcfs: no buildable Go source files in /home/kir/go/src/github.com/portworx/lcfs
kir@kd:~$ cd go/src/github.com/portworx/lcfs/testing
kir@kd:~/go/src/github.com/portworx/lcfs/testing$ go get ./...
# cd .; git clone https://github.com/portworx/px-test /home/kir/go/src/github.com/portworx/px-test
Cloning into '/home/kir/go/src/github.com/portworx/px-test'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
package github.com/portworx/px-test/graph: exit status 128

So, is there a way to reproduce the benchmarks you refer to in README?

This could become challenging on a fragmented file system.

"make test" does not report pass/fail for the tests run.

How feasible would be to build and ARM bundle?. Would really like to try this in a PI to see if it brings any improvements.

Hi,
When i use lcfs as docker storage driver, yum install some package will be stuck. I change devicemapper, start same image, install same package, it can install successfully. Why? I am not sure lcfs is available and feasible?

# yum install unzip
Loaded plugins: fastestmirror, ovl
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Resolving Dependencies
--> Running transaction check
---> Package unzip.x86_64 0:6.0-19.el7 will be installed

# lsb_release
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.4.1708 (Core) 
Release:	7.4.1708
Codename:	Core

# docker info
Containers: 5
 Running: 1
 Paused: 0
 Stopped: 4
Images: 5
Server Version: 18.06.1-ce
Storage Driver: portworx/lcfs
 Build Version: 1.0
 Library Version: 1.0
Logging Driver: json-file
Cgroup Driver: systemd

It seems as if the links to download and install lcfs in the Readme aren't working. Has the project been mothballed?

Currently, many things in LCFS are tracked using sequential lists (layers, files in small directories, dirty pages of a small file, extent map of a fragmented file, allocated extent map of layers, free extent map of the file system, extended attributes, hardlinks in a layer etc.). Switch to better data structures for these for better performance.

I ran latest LCFS (git commit 99b157d) and got into this while running "make" on docker daemon sources

# ./lcfs daemon /dev/sda /var/lib/docker /lcfs -f -m -r -t -p -v
.....
lcfs: fs.c:308: lc_removeLayers: Assertion `zfs->fs_super->sb_flags & LC_SUPER_INIT' failed.
Aborted (core dumped)

This is what I saw on the client:

Step 42/43 : ENTRYPOINT hack/dind
 ---> Running in 137aeada4f4d
 ---> 98b9d7df394f
Removing intermediate container 137aeada4f4d
Step 43/43 : COPY . /go/src/github.com/docker/docker
 ---> ba3aea4d5aa7
Successfully built ba3aea4d5aa7
open /var/lib/docker/image/portworx/lcfs/.tmp-repositories.json026490387: no such file or directory
Makefile:113: recipe for target 'build' failed
make: *** [build] Error 1

and here are dockerd logs:

INFO[9097] Layer sha256:d173fd590788d5c19e591594e8d6f28524d41626bbfb4d717228dc50db6b5674 cleaned up 
INFO[9104] time="2017-08-25T14:58:09Z" level=error msg="err software caused connection abort\n"   plugin=de91e35b7c8c14d0bb3190c519dd10eb0d5a5b88b4e78f91f635d4367c52f5b0
ERRO[9104] Error removing mounted layer 3ff6814c3cb6dc6e75732aa9738d29fd62bd7cc182b07f9050c5905a6a4d73a5: GraphDriver.Remove: software caused connection abort 
ERRO[9104] Failed to release RWLayer: GraphDriver.Remove: software caused connection abort 
ERRO[9104] failed to unmount previous build image sha256:2baf993621acd4dc745c97ff3127609dbf68ec73c9fedb8278fa029763d177f1: GraphDriver.Remove: software caused connection abort 
INFO[9104] time="2017-08-25T14:58:09Z" level=error msg="err transport endpoint is not connected\n"   plugin=de91e35b7c8c14d0bb3190c519dd10eb0d5a5b88b4e78f91f635d4367c52f5b0
ERRO[9104] Failed to unmount RWLayer: GraphDriver.Put: transport endpoint is not connected 
ERRO[9104] failed to unmount previous build image sha256:e1573ce8644fc2781917e5edf824b154f1492476f202b0f35340dc7119ad743f: GraphDriver.Put: transport endpoint is not connected 
INFO[9104] time="2017-08-25T14:58:09Z" level=error msg="err transport endpoint is not connected\n"   plugin=de91e35b7c8c14d0bb3190c519dd10eb0d5a5b88b4e78f91f635d4367c52f5b0
ERRO[9104] Failed to unmount RWLayer: GraphDriver.Put: transport endpoint is not connected 
ERRO[9104] failed to unmount previous build image : GraphDriver.Put: transport endpoint is not connected 
INFO[9104] time="2017-08-25T14:58:09Z" level=error msg="err transport endpoint is not connected\n"   plugin=de91e35b7c8c14d0bb3190c519dd10eb0d5a5b88b4e78f91f635d4367c52f5b0
ERRO[9104] Failed to unmount RWLayer: GraphDriver.Put: transport endpoint is not connected 
ERRO[9104] failed to unmount previous build image sha256:bdf6818dde703755f0a724e9a72579529fa66cdde79b1a5a9037184894dd6108: GraphDriver.Put: transport endpoint is not connected 
INFO[9104] time="2017-08-25T14:58:09Z" level=error msg="err transport endpoint is not connected\n"   plugin=de91e35b7c8c14d0bb3190c519dd10eb0d5a5b88b4e78f91f635d4367c52f5b0
ERRO[9104] Failed to unmount RWLayer: GraphDriver.Put: transport endpoint is not connected 
ERRO[9104] failed to unmount previous build image sha256:98b9d7df394f644af4a772e74a710f196aacd92806756881f797b2a98b52cb2e: GraphDriver.Put: transport endpoint is not connected 

Please let me know if there's anything else I can provide.

When lcfs is started, it should take in a size option and not exceed that capacity limit. Image layers that are not in use (perhaps using some sort of garbage collection scheme) should be automatically purged.

Invalidate pages when a file or a layer deleted instead of waiting cleaner to do that job.

LCFS displays its logs on stdout when run in daemon mode. There is no way to collect those when LCFS is run in daemon mode. Provide a method to collect LCFS logs on a demand basis.

As of now, all the metadata stays in cache always (inodes, directories, emaps, extended attributes etc.)

As of now, if system or lcfs crashes, the whole file system is wiped off (not when lcfs is cleanly unmounted).
Implement a scheme such that file system is intact even across abnormal shutdowns. A checkpointing scheme may be preferred over a journaling scheme.

After I reboot my system (ubuntu), Docker goes back to using AUFS as the driver. Consequently I lose my previous images.

LCFS needs to be retained across boots.

Hi, when i build lcfs following Manual installation. Starting lcfs should point device such as /dev/sdb. I point device, but get error

# lcfs daemon /dev/vdb /var/lib/docker /search/odin/lcfs
open: Device or resource busy
lcfs[19268]: Failed to open /dev/vdb

When i change a file, i get error

# lcfs daemon /search/odin/xuezhiyou/lcfs.dev /var/lib/docker /search/odin/lcfs
lcfs[25702]: Device is too small. Minimum size required is 40MB

# fdisk -l

Disk /dev/vda: 42.9 GB, 42949672960 bytes, 83886080 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00099a84

   Device Boot      Start         End      Blocks   Id  System
/dev/vda1   *        2048    83886079    41942016   83  Linux

Disk /dev/vdb: 859.0 GB, 858993459200 bytes, 1677721600 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x1ab19eeb

   Device Boot      Start         End      Blocks   Id  System
/dev/vdb2            2048      411647      204800   83  Linux

How should i do?

LCFS tracks many stats (like file system operations, memory usage, type of files etc). Also it supports enabling cpu profiling with gperftools. All these are enabled/disabled at build time. It would be good to provide an utility which can enable/disable these selectively at run time.

There are a few hash lists in use - for inodes in inode cache, pages in block cache, names in directory, dirty pages in a file etc. Improve those hashing schemes so that the distribution of those would more even with a random set of entries.

Hello,

I tried to install using the bash script and I get the following error. RUNNING IT AS ROOT

curl -fsSL http://lcfs.portworx.com/lcfs-setup.sh | DEV=/dev/sdc bash Unable to find image 'centos:latest' locally latest: Pulling from library/centos aeb7866da422: Pull complete Digest: sha256:67dad89757a55bfdfabec8abd0e22f8c7c12a1856514726470228063ed86593b Status: Downloaded newer image for centos:latest Preparing... ######################################## Updating / installing... lcfs-0.5.0-0 ######################################## [ ok ] Stopping docker (via systemctl): docker.service. Connecting to device, please wait... Note: LCFS device file exists. Using existing device file /dev/sdc without modifying. lcfs[12772]: Build: 3e2810a7e7150a231e03d9e75c565c330484cc13 Release: 0.5.0-3e2810a lcfs[12772]: Maximum memory allowed for data pages 802 MB lcfs[12772]: Formatting /dev/sdc, size 16106127360 lcfs[12772]: /dev/sdc mounted at /var/lib/docker lcfs[12772]: /dev/sdc mounted at /lcfs Error: failed to start docker. Unable to get the full path to root (/var/lib/docker): failed to canonicalise path for /var/lib/docker: lstat /var/lib/docker: transport endpoint is not connected

Also, tried installing it manually, but I´m also getting some errors. Under the "Build the lcfs file system" while executing make I get the following.

oot@docker-runner-110:/mnt/lcfs/lcfs# make gcc -g -msse4.2 -Wall -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse3 -I/usr/local/include/fuse3 -c -o cli.o cli.c In file included from /usr/include/x86_64-linux-gnu/urcu/wfcqueue.h:28:0, from /usr/include/x86_64-linux-gnu/urcu-call-rcu.h:35, from /usr/include/x86_64-linux-gnu/urcu.h:133, from includes.h:34, from cli.c:1: linux.h: In function ‘lc_lockOwned’: linux.h:42:43: error: ‘struct __pthread_rwlock_arch_t’ has no member named ‘__writer’; did you mean ‘__writers’? assert((lock == NULL) || lock->__data.__writer || ^ linux.h:43:40: error: ‘struct __pthread_rwlock_arch_t’ has no member named ‘__nr_readers’; did you mean ‘__readers’? (!exclusive && lock->__data.__nr_readers)); ^ linux.h:42:43: error: ‘struct __pthread_rwlock_arch_t’ has no member named ‘__writer’; did you mean ‘__writers’? assert((lock == NULL) || lock->__data.__writer || ^ linux.h:43:40: error: ‘struct __pthread_rwlock_arch_t’ has no member named ‘__nr_readers’; did you mean ‘__readers’? (!exclusive && lock->__data.__nr_readers)); ^ <builtin>: recipe for target 'cli.o' failed make: *** [cli.o] Error 1

I´m using gcc 7.6.

FUSE 3.0 library supports writeback caching and readdirplus.

Inode cache is traversed two times for every layer during unmount, combine those into one.

There is no global LRU maintained for block cache and that could turn out to be a problem for certain workloads. Pages are purged from block cache when usage is above limit, based on count tracking number of hits on it, but that does not account recently inserted pages. Figure out a better scheme possibly without adding more global locking.

Grow table of layers as new layers created rather than allocating for maximum supported number of layers at mount time (64K is the limit now).

Docker runs in a Linux VM on MAC. This VM could be accessed by running the following command:

"screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty"

It should be possible to install LCFS in that VM and start docker with LCFS as graphdriver. Provide a method to automate this for end user.

Investigate if return values from calls to fuse_reply_*() need to be checked, especially on interrupt.

Syncer takes checkpoints of LCFS frequently to keep LCFS crash consistent. This may trigger I/Os and loss of access to LCFS for short intervals. If somebody does not want LCFS to be crash consistent, they may disable syncer completely.

Each layer maintains a private cache of inodes. Size this based on the size of the dataset (meaning number of inodes in the layer).

LCFS does not honor UID/GID mappings provided by docker to the graphdriver.