OverlayFS about zram-config HOT 18 CLOSED

Your umount script it doing something wrong.
opt/$ZRAM_DEV is opt/overlay which is something that doesn't exist.
And the first umount $ZRAM_DIR already unmounts overlay.
If you provide the real $ZRAM_DEV instead of overlay source string in mount command you may get what you wanted.

Apart from that I did not fully understand the requirements to make changes to lower directory.
And change like that should be done when overlayfs is not mounted. I do not know of an exiting offline merge tool, but it will be good to have one.

from zram-config.

Hey your right doh! I hadn't really looked its the df as they have now no longer directly mounted to a zram device.
Apols and thanks.
Hows its found that!!! Because its in my code! :)
I am still trying to think of a merge without unmount as the unmount in one way or another is going to effect apps that run on them, some apps fail even if a log is unavail and would need individual restarts.
Guess just going to have to enforce a reboot or employ the devil of AUFS spin-locks.
I will sort the unmount script but the lack of an online merge is a pain for this application.
Its need is actually for very rare occurrences of new installs that may create directories and a system crash so that certain apps may just fail on a reboot if not there.
You could just write out on user input or a cron job a synctodisk without need of umount or stop.
But I guess I don't have to write to the lower and can write out to any persistent and just cp -arvfu to the lower before its mounted on boot.
In fact the latter and many thanks as just needed a break to have a think about it, would be great if you could do a merge with overlayFS though, even if it was slow to propagate through the mount.

from zram-config.

Right just be aware that cp may not sync deletes and renames in upper fs correctly to lower. You should test this. It will certainly not sync renamed directory correctly if redirect_dir option is enabled.
For these things a proper offline merge tool is needed.

from zram-config.

https://github.com/kmxz/overlayfs-tools?

Apols Amir but if on stop I cp -arvfu the mounted mergedir to lower then umount the unionfs that should be OK? Its just continuing to use as the copy will just screw the OverlayFS resultant merge.
The copy merged to lower then unload is OK?
I will have a read of your stuff at https://github.com/amir73il/overlayfs/wiki/Overlayfs-snapshots and stop bugging you :) many thanks and really appreciate the input.
As if the copy on stop is OK then the write outs without stop could be snapshots I guess.

from zram-config.

overlayfs-tools seems like a nice project. never tried. it does not seem to support redirect_dir,
but it does support opaque dirs and whiteouts, so better than cp.
You can always add support for redirect_dir...
You wont find my answers above in my wiki. You may bug me that's fine ;-)

from zram-config.

https://github.com/amir73il/overlayfs/wiki/Overlayfs-TODO#features
Offline/online layers merge tool? But no link.
What would you suggest?

from zram-config.

I would suggest trying out https://github.com/kmxz/overlayfs-tools. It supports the default overlayfs features and you can extend it to extra features if you need to.

from zram-config.

Ok some reading needed as my head is spinning slightly.
As Mr Schwarzenegger would say "I probably will"

from zram-config.

I did a bit of tidying and included overlayfs-tools which is great.

Just wondering can I enable redirect_dir via sys/module/overlay/parameters/* rather than edit /boot/cmdline.txt?

I am all ears about redirect_dir also wondering if snapshots could be used for what was the now deleted write method to call syncToDisk.
Currently its just on stop and my log in /usr/local/share/zram-config/log/zram-config.log looks fine and so does the resultant merge.

zram-config stop 2019-04-08-13:37:49
ztab remove log /zram2 /var/log /log.bind
/zram2
Warning: Stopping rsyslog.service, but it can still be activated by:
  syslog.socket
umount: /var/log (overlay) unmounted
overlay --lowerdir=/opt/zram/log.bind --upperdir=/opt/zram/zram2/upper
The script overlay-tools-Qkcisq.sh is created. Run the script to do the actual work please. Remember to run it when the OverlayFS is not mounted.
+ rm -rf /opt/zram/log.bind/auth.log
+ mv -T /opt/zram/zram2/upper/auth.log /opt/zram/log.bind/auth.log
+ rm -rf /opt/zram/log.bind/kern.log
+ mv -T /opt/zram/zram2/upper/kern.log /opt/zram/log.bind/kern.log
+ rm -rf /opt/zram/log.bind/daemon.log
+ mv -T /opt/zram/zram2/upper/daemon.log /opt/zram/log.bind/daemon.log
+ rm -rf /opt/zram/log.bind/wtmp
+ mv -T /opt/zram/zram2/upper/wtmp /opt/zram/log.bind/wtmp
+ rm -rf /opt/zram/log.bind/messages
+ mv -T /opt/zram/zram2/upper/messages /opt/zram/log.bind/messages
+ rm -rf /opt/zram/log.bind/syslog
+ mv -T /opt/zram/zram2/upper/syslog /opt/zram/log.bind/syslog
+ rmdir /opt/zram/zram2/upper
removed 'overlay-tools-Qkcisq.sh'
umount: /opt/zram/zram2/ unmounted
removed directory '/opt/zram/zram2'
umount: /opt/zram/log.bind/ unmounted
removed directory '/opt/zram/log.bind'
Warning: Stopping rsyslog.service, but it can still be activated by:
  syslog.socket
/dev/zram2 removed
ztab remove dir /zram1 /var/backups /backups.bind
/zram1
umount: /var/backups (overlay) unmounted
overlay --lowerdir=/opt/zram/backups.bind --upperdir=/opt/zram/zram1/upper
The script overlay-tools-iPXQBs.sh is created. Run the script to do the actual work please. Remember to run it when the OverlayFS is not mounted.
+ mv -T /opt/zram/zram1/upper/test /opt/zram/backups.bind/test
+ rm -rf /opt/zram/backups.bind/fart
+ mv -T /opt/zram/zram1/upper/fart /opt/zram/backups.bind/fart
+ rmdir /opt/zram/zram1/upper
removed 'overlay-tools-iPXQBs.sh'
umount: /opt/zram/zram1/ unmounted
removed directory '/opt/zram/zram1'
umount: /opt/zram/backups.bind/ unmounted
removed directory '/opt/zram/backups.bind'
/dev/zram1 removed
ztab remove swap /zram0 zram-config0 
/dev/zram0 removed
removed '/usr/local/share/zram-config/zram-device-list.rev'

Do you think snapshots will get included in the kernel? To be honest the `write' syncToDisk 'snapshot' function is not hugely important and from starting with log2ram always been slightly bemused as to why?
A snapshot or syncToDisk will always have a volatile gap to the previous that its probably better anyway to stop apps/services -> merge down to lower and restart.
But would be interested as some seem to think in this application it has value.

from zram-config.

You can change redirect_dir default via sys/module/overlay/parameters/
and you can enable it per overlay mount with mount option, see Documentation/filesystems/overlayfs.txt
for details.

overlayfs-tools does not seem to support redirect_dir. You might want to contact the author to ask about plans to support redirect_dir. I do not know the author and never used the tools myself.

I am not sure how overlayfs snapshots are related to syncToDisk functionality.
The feature is not rotating upper layer to lower layer if that is what you think.
Could be a nice feature, but I did not implement it.
Upstreaming plans are still quite far away at this point.

from zram-config.

mount --verbose --types overlay -o redirect_dir=on,lowerdir=${ZDIR}${BIND_DIR},upperdir=${ZDIR}/zram${RAM_DEV}/upper,workdir=${ZDIR}/zram${RAM_DEV}/workdir overlay ${TARGET_DIR}

Is now a mount option.
Asked kmxz in my noob manner about about rename/redirect_dir support.
Still slightly bemused at what would seem an almost total lack of any supporting tools for OverlayFS?
Are there any plans for maybe a Linux-utils addition for OverlayFS now it is a kernel module?

Yeah I was thinking maybe could mount as upper merge down then mount updated lower with zram upper on next boot (with snapshots rotate down that is).

from zram-config.

Thanks for this project btw. I'm also interested in finding an atomic copy_down and will keep tabs on the collective progress here!

I noticed you turned off redirect_dir in your latest commits, curious why? Did you run into specific problems with directory renames in lower?

from zram-config.

Purely the because the only tool for offline merge was written before that addition to OverlayFS.
To be honest unsure of the full implications but its to match the tools we have and that its choice if someone wishes to add it even though I think it is now default.

amir and the FS kernel file system guys know about the tool shortage as opposed to Aufs, but hoping for some suggestions but nothing seemed to be offered.

from zram-config.

has anyone considered using rsync ?

from zram-config.

has anyone considered using rsync ?

I agree with using rsync to sync.
I imagine like profile-sync-daemon.

from zram-config.

Thanks for this project btw. I'm also interested in finding an atomic copy_down and will keep tabs on the collective progress here!

I noticed you turned off redirect_dir in your latest commits, curious why? Did you run into specific problems with directory renames in lower?

Think it became a default or I have made a mistake, with it being default I decicided to leave out so implementers could decide.

https://github.com/amir73il is the man to ask about all this as active in the dev group.
I have been meaning to get back in touch and say hey overlayfs guys when you going to give us more tools like aufs?!

from zram-config.

@StuartIanNaylor you should address your question to containers developers, because they have the most to gain from overlayfs tools. I have been involved with implementing many overlayfs features in the kernel for the past couple of years (redirect_dir,index,nfs_export,xino). All of these features bring benefits to end users, but AFAIK, they are all still disabled by default in containers (e.g. docker) because lack of tools support (e.g. migration/import/export).
I cannot force anyone to use the new overlayfs features nor to write offline tools support for them.
I will post a message to the kernel developers list to ask how we can improve the situation.

from zram-config.

Thanks amir as its so frustrating as OverlayFS is brilliant but it does lack tools.
Really appreciate your input, many thanks.

from zram-config.