docker container (gitlab-ce) fails to start when using mergerfs as mounted data about mergerfs HOT 6 CLOSED

494   16:20:17.797566 newfstatat(AT_FDCWD, "/mnt/DISK/NytroWarpDrive/gitlab-ce/data/.ssh", 0x7f8d00b101a0, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied) <0.000030>
494   16:20:17.797655 newfstatat(AT_FDCWD, "/mnt/DISK/FlashMaxIII/gitlab-ce/data/.ssh", 0x7f8d00b101a0, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied) <0.000028>
494   16:20:17.797737 newfstatat(AT_FDCWD, "/mnt/SAN/NAS_LUN-1/gitlab-ce/data/.ssh", 0x7f8d00b101a0, AT_SYMLINK_NOFOLLOW) = -1 EACCES (Permission denied) <0.000029>

Because you don't have a strace of the offending app I can't really correlate things properly but this certainly looks like a problem. The underlying filesystems are returning perm denied for that file / path. Have you confirmed perms are properly set?

from mergerfs.

Ah I misread that support step and only gave the stacktrace gitlab generated internally.
Here is the strace from app command gitlab-ctl reconfigure since I'm not sure which is the most basic command is failing. This is just a snippet from where i see the error begin until the end. Let me know if you need the whole file (90M)
snippet.app.strace.txt

I don't think folder permissions are the issue because these are all folders that the container is creating and setting of permissions itself. Their script is here: update-permissions

Further testing if i swap the volumes in my compose to use /mnt/DISK/NytroWarpDrive directly instead of the /mnt/data pool, the container starts up normally without error. Issue only exists when using pool as a mount

version: '3'
services: 
  gitlab-ce:
    container_name: GitLab-CE
    image: 'gitlab/gitlab-ce:15.4.6-ce.0'
    ports:
      - '9081:80'
    volumes:
        - '/mnt/DISK/NytroWarpDrive/gitlab-ce/config:/etc/gitlab'
        - '/mnt/DISK/NytroWarpDrive/gitlab-ce/data:/var/opt/gitlab'
        - '/mnt/DISK/NytroWarpDrive/gitlab-ce/log:/var/log/gitlab'`

Other notes: NytroWarpDrive is the only disk with the gitlab-ce folders. This is a fresh vm install which is why disk sizes are small and empty.

from mergerfs.

From mergerfs' perspective the OS is absolutely returning permission denied. It is there in the mergerfs strace as I shared. It stat'ed the .ssh path and all three mounts returned EACCES.

I need both the strace from mergerfs and the strace from the app at the same time so I can correlate what request the app sends with the behavior of mergerfs.

Issue only exists when using pool as a mount

Yes, but perms can be different due to how containers work and how you have mergerfs setup. For instance: many people don't share groups between a container and host leading to a supplemental group difference. Some users use user namespacing which further changes what is going on between a container and the host. You could have perm errors on parts of the path that don't translate to a bind mount. Not everything can be exactly replicated between mergerfs and the underlying filesystem. Hence why I need all the information possible about the setup to comment. Almost certainly the issue is permissions... but I need to know what they are.

from mergerfs.

This is a fresh vm install

So what are the perms of the /mnt/DISK/* ? Are you positive they setup properly? mergerfs sees /mnt/DISK/foo... not what you bind mount. It evaluates not the binded point down. It looks at the whole path. If the base of the path is not properly permissioned such that from outside the container it works then it won't work in the container either.

from mergerfs.

D'oh! You were right it was a permission issue. I went over the folder permissions earlier before opening this issue but i must have misread the output. Specifically /mnt/DISK had 750 for permission. Once i changed it to 755 and recreated the gitlab container the startup ran smoothly without error. Sorry to waste your time. I swear i checked all this before hand.

Thank you for your ongoing support and overall awesome projects. Long time scorch user, but still a mergerfs newbie.

from mergerfs.

no problem. glad we resolved it.

This is a somewhat complicated situation. If I rewrote mergerfs to work more like what a bind mount would do it would 1) require a major rewrite and 2) means that branches can't as easily be added and removed without explicitly configurating mergerfs because it would require holding an open file the life of the usage. It might be worth doing but need to carefully consider all the consequences.

from mergerfs.