Trying to balance throws the error "is not a mergerfs mount" about mergerfs HOT 16 CLOSED

xattr=nosys

You have specificly disabled xattr so the runtime api is disabled.

from mergerfs.

No, it is more performant in the sense that xattr calls would be disabled and so naturally less work means less overhead. And with page.cache enabled it can lead to the kernel issuing getxattr calls to find out certain information about files way more than normal. But the runtime API uses xattr so if you turn it off so does the runtime api get turned off. I mention it in the docs. Until I change the method of communicating with mergerfs to use ioctl or a different mechanism there isn't much I can do. It is planned but rarely an issue for people so lower priority.

from mergerfs.

do you mean I can have two mergerfs pools with the exact same volumes under them?

Yes.

from mergerfs.

That's amazing, I'll keep that in mind for any future trouble I want to get myself into.

For now though, thank you again for helping me with this.

from mergerfs.

This is probably more appropriately registered to the tools repo but... could you provide the details requested in the ticket template? I don't know what your mergerfs settings are. The tools in mergerfs-tools require xattr enabled because that's how the runtime interface works currently. As we see in the trace of mergerfs.balance its not finding it.

256059 16:13:17.759263 lgetxattr("/srv/mergerfs/merged/.mergerfs", "user.mergerfs.version", 0x7f2b31edfeb0, 64) = -1 EOPNOTSUPP (Operation not supported) <0.000104>

from mergerfs.

Understood, if I need to move this post I don't mind.

My bad, I completely missed the config.
This is what I see in the file

/etc/systemd/system/srv-mergerfs-merged.mount

[Unit]
Description = MergerFS mount for merged
After=network-fs.target zfs-mount.target
RequiresMountsFor=/srv/dev-disk-by-uuid-0cdf8cd4-5c49-460a-a65a-6e340b53a2a4 /srv/dev-disk-by-uuid-5712d8c8-09ba-41d8-b9b7-798642b87770 /srv/dev-disk-by-uuid-5a8ed703-605e-48ec-a507-43e17f0c3504 /srv/dev-disk-by-uuid-907a124e-7a2b-4b94-8dbf-a79179e25a0c /srv/dev-disk-by-uuid-913a945f-428d-43ab-987d-eebeb24f7161 /srv/dev-disk-by-uuid-b90366a7-6fea-4a00-b18c-0b67ce756d97 /srv/dev-disk-by-uuid-c67fd0d7-2ed4-4e00-98dd-c1a7c168e5bd /srv/dev-disk-by-uuid-d5a72cf7-8b04-4475-9860-9c48735b3f23 /srv/dev-disk-by-uuid-c1051291-a5bf-42e6-93a9-8d541f08aebb /srv/dev-disk-by-uuid-67fd7113-273a-4219-bd3b-43e69df1dbb2 /srv/dev-disk-by-uuid-98049d61-1c9b-4ec1-a698-0c2c1308a573 /srv/dev-disk-by-uuid-e0985793-e695-4008-8cc5-255be3783ebb /srv/dev-disk-by-uuid-d2a25710-050e-4584-9eb8-eb1c9661433a /srv/dev-disk-by-uuid-03db52c8-c739-494b-b96f-37f53354b185

[Mount]
What = merged:e635f518-3fba-469c-8719-d0b3862f09f2
Where = /srv/mergerfs/merged
Type = fuse.mergerfs
Options = branches=/srv/dev-disk-by-uuid-0cdf8cd4-5c49-460a-a65a-6e340b53a2a4:/srv/dev-disk-by-uuid-5712d8c8-09ba-41d8-b9b7-798642b87770:/srv/dev-disk-by-uuid-5a8ed703-605e-48ec-a507-43e17f0c3504:/srv/dev-disk-by-uuid-907a124e-7a2b-4b94-8dbf-a79179e25a0c:/srv/dev-disk-by-uuid-913a945f-428d-43ab-987d-eebeb24f7161:/srv/dev-disk-by-uuid-b90366a7-6fea-4a00-b18c-0b67ce756d97:/srv/dev-disk-by-uuid-c67fd0d7-2ed4-4e00-98dd-c1a7c168e5bd:/srv/dev-disk-by-uuid-d5a72cf7-8b04-4475-9860-9c48735b3f23:/srv/dev-disk-by-uuid-c1051291-a5bf-42e6-93a9-8d541f08aebb:/srv/dev-disk-by-uuid-67fd7113-273a-4219-bd3b-43e69df1dbb2:/srv/dev-disk-by-uuid-98049d61-1c9b-4ec1-a698-0c2c1308a573:/srv/dev-disk-by-uuid-e0985793-e695-4008-8cc5-255be3783ebb:/srv/dev-disk-by-uuid-d2a25710-050e-4584-9eb8-eb1c9661433a:/srv/dev-disk-by-uuid-03db52c8-c739-494b-b96f-37f53354b185,category.create=mfs,minfreespace=4G,fsname=merged:e635f518-3fba-469c-8719-d0b3862f09f2,defaults,allow_other,cache.files=auto-full,use_ino,dropcacheonclose=true,xattr=nosys,nonempty

from mergerfs.

I have to wait a bit before I can apply the change. I'll close this once I've verified it works. Thank you!

from mergerfs.

@trapexit Verified that works, thank you again. I turned xattr off because based on the readme it seems like off is the more performant option. Did I misread that?

from mergerfs.

Got it, so for normal use I can disable xattr and gain that bit of performance, but if I want to use any of the tools I need to re-enable xattr.

from mergerfs.

Yes, but unfortunately it isn't that simple at runtime. If you tell the kernel that xattr isn't supported (by telling mergerfs to use nosys errors) then the kernel caches that for the lifetime of the mount. There is no way for mergerfs to tell the kernel "hey... I randomly support xattr now". It is all or nothing and once it is told it isn't supported it never will send another xattr request till restarted. Nothing I can do about that.

The tool could be rewritten not to use the runtime api but that would require people to manually manage the list of required fields. I think for balance it is just the list of branches... but really I always question people needing the balance tool as it is really big hammer solution for something that many people tend to want more nuance for. Or they screwed up their settings to begin with.

from mergerfs.

I've had xattr off for at least a year, balance seems to be working though. I don't have any qualms with turning it off and on manually since I've only used balance twice in the ~4 years I've been using mergerfs. In both cases it was because I added a brand new drive to the array and I wanted to make sure mergerfs continued to place files across all drives. I have mergerfs configed to use the drive with the most free space.

In this case I had to restart OMV for the xattr change to be recognized and doing that is not an issue for me either.

Would that be considered a misuse of the tool, or am I doing something wrong there? I'm worried about data loss more than anything else.

from mergerfs.

I don't know exactly what you're looking to do so it is difficult to comment. You're looking to move data to the new drive assuming that newer means less likely to fail and want to move some data to the new filesystem to spread out risk? And you don't want to wait for balancing to happen naturally?

If so, for the purpose of data loss, I'm not sure this makes much sense. Prediction of failure is hard/impossible and you should just have backups of the data you need to be able to restore anyway which should negate any setup concerns except if you do something like a mergerfs pool that includes RAID setups and use epmfs or similar to clearly target specific branches.

The only real reason I see for spreading out files is performance and that can also be accomplished by using rand policy so the new branch doesn't receive all new files. pfrd would be second best for that I think.

from mergerfs.

Thank you for talking me through this, I appreciate it a lot.

Most of my drives are 4TB and mostly full. I've been upgrading them to 8TB, and when I do that I end up with a mostly empty drive. It bothers me to see one drive that's mostly empty , and with my current config mergerfs will use that new drive heavily until its used up as much as the others.

The worry about data loss was related to turning attr off then on again before doing balance.

But seems like I should change my mergerfs config to use a different create policy, right now I'm using most free space.

I'm going to let the balance finish what its doing, and then take some time to really understand the create policies better and move away from most free space. I agree rand sounds like the best option for me with percent free rand as the second best choice. Once I do that I shouldn't need to do a manual balance.

from mergerfs.

unless you are using software that gets confused by xattr existing or not and does something weird when it happens then no... there is no risk of data loss

At the end of the day the perf of the system in this context is related to access patterns. There isn't much mergerfs or anything else can do as it can't predict the future. Unless you specifically know the pattern of access there is nothing you can practically do. If you want to keep your branches filled at similar levels and don't have much churn in your system then yes, you will need out of band replacement of files but that replacement won't do anything for perf necessarily. It will spread files around but without knowing the access patterns it could be worse. All this stuff is very contextual. If the concern is literally just "It bothers me to see one drive that's mostly empty" you may want to ask why. Is there (practical) functional or risk differences or is it just some OCD thing. I'm not saying it is for you but a lot of people over the years have had reasons that boiled down to "just because that's how I want it." There absolutely are some negative consequences if a new drive is given all new files but are you hitting them and what can be done about them? Don't forget you could simply create a new pool with xattr enabled and just schedule mergerfs.balance or a custom app to move stuff around as you wish out of band on a regular basis.

from mergerfs.

I don't think any software I use on that array cares about xattr except for in this case the mergerfs tools.

The array is read heavy compared to writes, and if I'm understanding the role of the xattr setting in mergerfs the main performance issue is related to lots of writes. So in my use case the performance difference would probably be minimal if noticeable at all.

You're correct that the issue here is "It bothers me to see one drive that's mostly empty" and is nothing more than my own discomfort in seeing a drive that's not at the same use level as the rest.

"Don't forget you could simply create a new pool with xattr enabled " do you mean I can have two mergerfs pools with the exact same volumes under them?

I feel like changing the create policy to random will solve my issue, and I don't think balancing is something I'll do in the future. It puts unnecessary stress on all the drives and at the end of the day I can just ignore the one out of balance drive until the whole array naturally balances out.

from mergerfs.

mergerfs is just a proxy. People use the underlying filesystems out of band all the time.

https://github.com/trapexit/mergerfs#can-filesystems-be-written-to-directly-outside-of-mergerfs-while-pooled

The suggested way I have in all docs for "evacuating" a branch is to mount a replica of the pool minus the branch in question and rsync from the filesystem to the temporary pool.

from mergerfs.