Comments (4)
trapexit
commented on June 12, 2024
1
Oh, right. I forgot that FUSE does not support FICLONE. I don't even get the request from the kernel. As I understand the kernel calls the "file_operation" call clone_file_range when FICLONE is issued... which FUSE does not define.
Until the kernel supports this FICLONE and FICLONERANGE just aren't going to work. I can fire off an email to the FUSE mailing list and see what they think.
Also... if the kernel just gave me the source file descriptor it wouldn't be sufficient. That value comes from the client app. It would have no meaning to mergerfs. Maybe I could work around it by looking up the FD in /proc of the calling app. Not elegant might work.
from mergerfs.
BujiasBitwise
commented on June 12, 2024
1
I confirm that FICLONE is working with link_cow just fine, and my trace looks like yours. The problem in my previous test was that I truncated the file by mistake, and when I saw there were no shared extents I thought they hadn't been reflinked.
Regarding FICLONE, it's unfortunate FUSE doesn't support it. Maybe they have a good reason for that. Reflinks are a valuable feature, so it would be nice if they worked.
Only you can determine if going through the effort of ugly hacks is worth it. At least people can now take a look at this issue to see why it's not implemented. Or maybe add a simple comment to the readme, where you mention hardlink support.
I'll use link_cow for now, because I don't need hardlinks in my use case. Or I may just create them out of band, since it's much faster (0.5s vs 12s for 52k files)
Thanks for looking into this.
from mergerfs.
trapexit
commented on June 12, 2024
FICLONE got disabled when I had to blacklist all BTRFS ioctl calls a while back due to them being incompatible with how ioctl works via FUSE and due to my open policy prior it could cause crashes. I can whitelist that one call because that is safe but it was just an oversight at the time. I've been meaning to go through a bunch of popular ioctl calls to see if I can support them.
As for ficlone not working with link_cow... are you positive? I just tried it and it works fine. From a trace of mergerfs with a linked file opened for write.
[pid 2339892] newfstatat(AT_FDCWD, "/tmp/xfs/foo", {st_mode=S_IFREG|0644, st_size=1048576, ...}, AT_SYMLINK_NOFOLLOW) = 0
[pid 2339892] openat(AT_FDCWD, "/tmp/xfs/foo", O_RDONLY|O_NOFOLLOW) = 6
[pid 2339892] openat(AT_FDCWD, "/tmp/xfs/.MFtGOyCkOTYiEnZl", O_WRONLY|O_CREAT|O_EXCL|O_TRUNC, 0200) = 7
[pid 2339892] newfstatat(6, "", {st_mode=S_IFREG|0644, st_size=1048576, ...}, AT_EMPTY_PATH) = 0
[pid 2339892] ftruncate(7, 1048576) = 0
[pid 2339892] ioctl(7, BTRFS_IOC_CLONE or FICLONE, 6) = 0
[pid 2339892] ioctl(6, FS_IOC_GETFLAGS, [0]) = 0
[pid 2339892] ioctl(7, FS_IOC_SETFLAGS, [0]) = 0
[pid 2339892] flistxattr(6, NULL, 0) = 0
[pid 2339892] futex(0x7fa8c1a29788, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid 2339892] fchown(7, 0, 0) = 0
[pid 2339892] fchmod(7, 0100644) = 0
[pid 2339892] utimensat(7, NULL, [{tv_sec=1706395088, tv_nsec=16974695} /* 2024-01-27T16:38:08.016974695-0600 */, {tv_sec=1706395084, tv_nsec=408948479} /* 20
24-01-27T16:38:04.408948479-0600 */], 0) = 0
[pid 2339892] rename("/tmp/xfs/.MFtGOyCkOTYiEnZl", "/tmp/xfs/foo") = 0
[pid 2339892] close(6) = 0
[pid 2339892] close(7) = 0
I could reorder the ftruncate to be after the ficlone as I don't think it is necessary but afaict it is working as expected.
from mergerfs.
chapmanjacobd
commented on June 12, 2024
I've also needed this at times and I wrote a small wrapper around cp to resolve paths to the original drive. Might not fit every use-case but thought that I would share:
$ pip install xklb xattrs
$ lb mergerfs-cp --dry-run d/files* d/folder2/
cp --interactive --reflink=always /mnt/d9/files1.txt /mnt/d9/folder2/files1.txt
cp --interactive --reflink=always /mnt/d3/files1.txt /mnt/d3/folder2/files1.txt
...
$ btrfs fi du /mnt/d3/files1.txt /mnt/d3/folder2/files1.txt
Total Exclusive Set shared Filename
12.57GiB 0.00B 12.57GiB /mnt/d3/files1.txt
12.57GiB 0.00B 12.57GiB /mnt/d3/folder2/files1.txt
- https://github.com/chapmanjacobd/library/blob/main/xklb/folders/mergerfs_cp.py
- https://github.com/chapmanjacobd/library/blob/main/tests/folders/test_mergerfs_cp.py
from mergerfs.
Related Issues (20)
- Merge of directory level
- MergerFS mount randomly disappears, only displays ??? when listed HOT 66
- Operation not permitted when mounting as non-root HOT 5
- Failing to mount from fstab on boot, but can mount from command line HOT 5
- ubuntu 18.04 fresh install, with error HOT 1
- Permissions issue qBittorrent file creation when preload library is used in docker instance HOT 9
- unable to create new files (touch: cannot touch 'test': Not a directory) HOT 6
- Hangs when overlayfs is mounted on top of mergerfs's branch HOT 7
- Mergerfs volume with a single disk pool HOT 2
- Is it possible to use 'mount -a' to rebuild a mergerfs pool that is defined in fstab? HOT 6
- Tiered caching clarification HOT 7
- mergerfs - fuse `allow_other` and security HOT 3
- Benchmarking tool HOT 3
- fluctuating write performance HOT 13
- "&" sign in dirs path HOT 3
- The Debian12 mergerfs ls&df command displays an exception. HOT 7
- FUSE passthrough HOT 3
- Android smb clients issue on remove files. HOT 4
- How can I avoid moving files across file systems (mechanical hard disks) caused by Path Preservation? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mergerfs.